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ABSTRACT 


The Chief of Naval Operations defines FORCEnet as the 
"operational construction and architectural framework for 
Naval Warfare in the Information Age which integrates 
warriors, sensors, networks, command and control, platforms 
and weapons into a networked, distributed combat force, 
scalable across the spectrum of conflict from seabed to 
space and sea to land." The Trident Warrior experiments 
are the Navy's premier FORCEnet Sea Trial experiments. The 
purpose of the Trident Warrior experiments is to provide 
speed to capability and to develop supporting tactics, 
techniques, and procedures. Speed to capability provides a 
rapid fielding of improved capabilities to the fleet. The 
supporting tactics, techniques and procedures optimize the 
employment of the new technology. 

The purpose of this thesis will be to provide a basic 
overview of the Trident Warrior Experimentation Process. 
Through a step-by-step analysis, this thesis will explain 
and justify the individual steps that comprise a successful 
experiment/experimentation campaign. 
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I. 


INTRODUCTION 


A. SEA POWER 21 

In October 2002, Admiral Vernon Clark, the Navy's 
twenty-seventh Chief of Naval Operations (CNO), laid out 
his vision for the Navy of the future in Sea Power 21. 
Adm. Clark foresees, "Future naval operations [that] will 
use revolutionary information superiority and dispersed, 
networked force capabilities to deliver unprecedented 


offensive 

power. 

defensive 

assurance. 

and 

operational 

independence to Joint 

Force 

Commanders" 

(Sea 

Power 21). 

The three 

pillars 

of 

Sea Power 21 are 

Sea 

Strike, Sea 


Shield, and Sea Basing. 


SEA POWER 21 

Sea Shield 



Sea Basing 


Figure 1. Sea Power 21(From: Sea Power 21) 


Sea Strike is the ability to inflict a persistent and 
precise offensive force. In order to provide this 
offensive power. Sea Strike must be capable of persistent 
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intelligence, surveillance, and reconnaissance; time 
sensitive strikes; electronic warfare and information 
operations; ship-to-objective maneuver; and covert strike. 
Sea Strike will provide an amplified, effects-based 
striking power; increase effectiveness of precision 
attacks, enhance the war fighting contribution of Marines 
and Special Forces, and allow seamless integration of joint 
strike packages. The capabilities associated with Sea 
Strike will continue to grow with technological advances in 
unmanned combat vehicles, hypersonic missiles, electro¬ 
magnetic rail guns, and a vast array of sensor 
improvements. 

Sea Shield provides defensive assurance to our forces 
and our allies across the globe. Sea Shield will provide 
layered, global defensive power based on control of the 
seas, a forward presence, and networked intelligence, in 
order to enhance homeland defense, assure littoral access, 
and project power inland. Future technologies within Sea 
Shield include integrating existing technology into a 
common undersea picture and a single integrated air 
picture, intelligence and communications reach-back 
systems, theater missile defense, organic mine 
countermeasures, and autonomous unmanned vehicles. Ideally 
these technologies will expand combat reach, create a 
common operating picture, and improve self-defense 
capabilities to ensure sea superiority. 

Sea Basing provides operational independence to naval 
forces, as well as support for the joint force. Sea Basing 
will provide pre-positioned war fighting capabilities, 
enhance joint support for a dispersed naval force, increase 
force security and operational agility, and minimize 
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reliance on shore infrastructure. Sea Basing must provide 
positioning afloat, offensive and defensive power 
projection, logistical and command and control (C2) 
support, and accelerated deployment and employment 
timelines for joint assets. In order to meet these 
challenges. Sea Basing will need increased capabilities to 
move heavier cargo faster, the ability to access all cargo 
while at sea, and enhanced and integrated Joint C2 and 
logistics capabilities. 

Sea Power 21 implements a Global Concept of Operations 
(GCO) that disperses combat power through a variety of 
platforms that possess an immense warfighting capability. 

This will allow us to "dissuade, deter, and defeat both 
regional adversaries and trans-national threats" ( Sea Power 
. 

While Sea Strike, Sea Shield and Sea Basing are the 

three pillars of Sea Power 21, a triad of organizational 
processes. Sea Warrior, Sea Enterprise, and Sea Trial, will 
be used to align and accelerate the development and 
incorporation of new technology for the fleet. Sea Warrior 
will invest in the Navy's human capital. Today's Navy 
employs cutting edge technology, requiring naval personnel 
to be both highly skilled and highly trained. Sea Warrior 
will provide Sailors who are skilled, motivated, and 

properly employed, producing a more effective, more 
efficient fleet. Sea Enterprise will incorporate modern 
business practices to reduce costs and improve efficiency. 
Sea Enterprise seeks to further improve efficiency by 

seeking opportunities to work with other services on common 
goals. Joint operations save money, improve inter¬ 

operability, and promote system integration. 
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The final process designed to implement Sea Power 21 
more smoothly and effectively is Sea Trial. Sea Trial is a 
fleet-led process of concept and technology development 
that delivers enhanced capabilities to the fleet. Sea 
Trial will coordinate the fleets, technology development 
centers, and academic resources to integrate wargaming, 
experimentation, and exercises that will increase 
warfighting capabilities. 

We often cite asymmetric challenges when 
referring to enemy threats, virtually assuming 
such advantages belong only to our adversaries. 

"Sea Power 21" is built on a foundation of 
American asymmetric strengths that are powerful 
and uniquely ours. Among others, these include 
the expanding power of computing, systems 
integration, a thriving industrial base, and the 
extraordinary capabilities of our people, whose 
innovative nature and desire to excel give us our 
greatest competitive advantage. ( Sea Power 21 ) 

B. FORCENET 

FORCEnet is the method in which the U.S. Navy will 
build on its asymmetrical strengths. FORCEnet is designed 
to be "the operational construct and architectural 
framework for naval warfare in the information age, 
integrating warriors, sensors, command and control, 
platforms, and weapons into a networked, distributed combat 
force" ( FORCEnet: A Functional Concept ). FORCEnet is often 
described as the 'glue' that holds Sea Power 21 together. 
FORCEnet will improve the speed and accuracy of the 
decision-making process, while integrating knowledge and 
providing a common operating picture to dominate the 
battlespace. This will ultimately allow commanders to make 
timelier, better-informed, and more accurate decisions. By 
increasing accessibility to information, FORCEnet will 
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create a network effect, exponentially increasing the value 
of information or capabilities provided by the network. 
FORCEnet will provide multi-tiered weapons grids; 
survivable networks; automated decision aids; and 
distributed, collaborative C2, while maintaining human¬ 
centric integration. FORCEnet will continue to promote a 
highly adaptive and decentralized form of command and 
control, empowering unit commander's initiative, 
adaptability and increased tempo, while maintaining the 
coordination of unity of effort that is provided by 
centralization. 

We have been talking about network-centric 
warfare for a decade, and EORCEnet will be the 
Navy's plan to make it an operational reality. 
Supported by EORCEnet, Sea Strike, Sea Shield, 
and Sea Basing capabilities will be deployed by 
way of a Global Concept of Operations that widely 
distributes the firepower of the fleet, 
strengthens deterrence, improves crisis response, 
and positions us to win decisively in war. (Sea 
Power 21) 


C. TRIDENT WARRIOR 

Trident Warrior (TW) is the EORCEnet component to Sea 
Trial. It is an annual event that focuses on pairing 
concepts and technology to provide enhanced capabilities to 
the fleet as rapidly as possible. The Naval Network 
Warfare Command (NETWARCOM) is the Sea Trial Operational 
Agent for FORCEnet, and is responsible for prioritization, 
coordination, validation, and oversight of all events 
within Trident Warrior (TW) . NETWARCOM is the operational 
agent (OA), and the Space and Naval Warfare Systems Command 
(SPAWAR), is the chief engineer for the Trident Warrior 
experiments. The experiments seek to improve EORCEnet 
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Command and Control capabilities by providing the 
warfighter with information superiority over an adversary, 
which would allow the warfighter to make decisions and 
execute commands more efficiently. By providing these 
capabilities, along with full supportability and 
maintainability. Trident Warrior underwrites success in the 
battlespace. 

Trident Warrior seeks to provide or improve upon many 
of the aforementioned capabilities of FORCEnet, including: 
multi-tiered sensor and weapon information; distributed, 
collaborative command and control; dynamic, multi-path and 
survivable networks; automated decision aids; and human¬ 
centric integration. Trident Warrior evaluates new 
technologies and approaches to resolving fleet needs using 
fleet experimentation and exercises. This is done in a 
spiral development process that enables the delivery of the 
new capability to be synchronized with the delivery of the 
required policies; tactics, techniques, and procedures 
(TTP); and concept of operations (CONORS). Trident Warrior 
measures the benefits of proposed technology and 
demonstrates its real capabilities. 

The end state of a Trident Warrior experiment is a 
Military Utility Assessment (MUA) that will provide a 
recommendation for advancements or changes in Doctrine, 
Organization, Training, Materiel, Leadership, Personnel, 
and Facilities (DOTMLPF). 

D. FORCENET INNOVATION AND RESEARCH ENGINE 

In order to facilitate the rapid dissemination of 
data, the Trident Warrior team utilizes the FORCEnet 
Innovation and Research Engine (FIRE website). 
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"FIRE was developed out of the need for 
structured data collection, data reconstruction 
for analysis and generation of TW analysis 
reports. No such system previously existed, and 
the Naval Postgraduate School (NPS) the analysis 
lead for TW, was asked to examine different 
approaches. NPS developed FIRE as an enterprise 
computing solution, based on Oracle 9i and Oracle 
lOg technology, with unique artificial 
intelligence (AI) applications included in the 
design." (Poeltler and Gallup, 30) 

FIRE has become a very useful tool for the Trident 
Warrior team. Access to FIRE allows the team to share both 
proprietary and confidential information, monitor the 
status of major objectives, and consult major reference 
publications that are written in conjunction with the 
Trident Warrior experiments. 

E. TRIDENT WARRIOR EXPERIMENTATION PROCESS 

Every Trident Warrior experiment begins with an 
overarching concept. This overarching concept is the 
general theme of the experiment. Recent themes of Trident 
Warrior Experiments include the Global War on Terrorism 
(GWOT) and the Command and Control (C2) of an Expeditionary 
Strike Group (ESC). Several key initiatives that will make 
significant contributions to the warfighter within this 
overarching concept are then identified. Each initiative 
is then dissected into multiple objectives that are 
essential to initiative development. Goals that are needed 
to realize the objectives are identified. The 
experimentation team then describes attributes that the 
proposed FORCEnet solution will need to meet each goal. 
Finally, the experiment is executed to compile data that 
will validate the attributes. 
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Anatomy of an Experiment 
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Figure 2. Breakdown of an Experiment 
In order to promote successful experimentation. 
Trident Warrior has established the Trident Warrior 
Experimentation Process. By following the steps of this 
process in order and within the prescribed timeframe, the 
Trident Warrior team has maximized the potential for 
success within and experimentation campaign. The steps of 
the Trident Warrior Experimentation Process are: 

1. Establish Team 

2. Concept Development 

3. Technology/Tactics, Techniques, and Procedures (TTP) 
Harvest 

4. Asset Identification 

5. Develop Experiment Objectives 

6. Integrated Definition (IDEE)/ Operational Sequence 
Diagrams (OSD) /Process Action Maps 

7. Experiment Design 

8. Event Definition 
Data Collection Plan 


9. 












































10 . 

Execution 



11. 

Final Report 



12 . 

Assessment Operational 

Agent 

Assessment (OAA) 

13. 

Military Utility Asses 

sment 

(MUA). 
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II. ESTABLISHING A TEAM 
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Figure 3. Establish Team 


An experimentation campaign is a major undertaking 
that will require various skills and expertise. Many 
individuals will be required to perform the numerous tasks 
required for success. Every individual will bring with 
them their own skill sets, personality traits, and ideas on 
how they believe the experiment should be run. Balancing 
the needs and desires of each individual, while ensuring 
that the group reaches its objective, can be a daunting 
task. In order to conduct a successful experiment, these 
individuals must refocus and realign their priorities to 
become a team. 
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A. WHAT IS A TEAM? 

"A team is a small number of people with complimentary 
skills who are committed to a common purpose, performance 
goals and an approach for which they hold themselves 
mutually accountable" (Katzenbach and Smith, 1994). 

1. Small Numbers 

According to a study conducted by Katzenbach and Smith 
in 1994, most productive teams consist of a small number of 
people, usually between 2 and 25 people. The most highly 
effective teams usually consisted of less than 10 people. 
Teams comprised of a larger number of people experience 
difficulty in interacting constructively as a group. Large 
teams also have even more problems trying to agree on 
specific steps that should be taken. 

2. Complimentary Skills 

The fundamental tenant of teamwork is that the 
combined efforts will be greater than the sum of the 
individuals working alone. In order to maximize the team 
efforts, it is necessary that teams be comprised of members 
with varying skills sets. In general, there are three 
types of basic skills sets: technical skills, problem 
solving/decision-making skills, and interpersonal skills. 
A technical skill is a functional expertise within a 
specific area, such as marketing or engineering. A problem 
solving/decision-making skill can be used to identify 
problems and opportunities, evaluate options, and make the 
necessary trade-offs. There are a variety of interpersonal 
skills, including "risk taking, helpful criticism, 
objectivity, active listening, giving the benefit of the 
doubt, support, and recognizing the interests and 
achievement of others" (Katzenbach and Smith, 1994). It is 
important to note that both problem solving/decision-making 

12 



skills and interpersonal skills can be developed and 
enhanced throughout the life of the team. 

3. Common Purpose and Performance Goals 

True teams can be identified by their strong desire to 
achieve some greater common purpose. A common, meaningful 
purpose will set the tone and direction for the team. The 
team's common purpose should broadly frame the requirements 
and define the boundaries and scope clearly enough to keep 
the team on track. However, it must also be flexible 
enough to allow the team to evolve. This broad directive 
can then be narrowed into specific performance goals. By 
creating specific performance goals, the team will be able 
to track their progress: either the task is complete or 
incomplete; the objective was met or not met. 

4. Common Approach 

By uniting in a common approach, all team members stay 
on track and their combined efforts are maximized. A 
team's common approach should identify what each team 
member's job will be, how schedules will be set and adhered 
to, how the team will agree upon and modify decisions, and 
how to determine the best approach for getting the job 
done. In addition to these steps, most teams have members 
who will evolve into the roles of: "challenging, 

interpreting, supporting, integrating, remembering, and 
summarizing. These roles will help promote mutual trust 
and constructive criticism," (Katzenbach and Smith, 1994) 
within the team. 

5. Mutual Accountability 

In order to be truly effective, a team must hold 
itself accountable. Forced accountability from a 

supervisor usually works best with individual efforts, but 
self-accountability within a team will promote commitment 
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and trust. Each team member must keep in mind that if one 
person fails, the team fails. 

B. TYPES OF TEAMS 

1. Purpose Classification 

One way to classify teams is by their purpose within 
an organization. Three common teams within this area 
include: 

a. Problem Solving Teams 

Problem solving teams are usually comprised of 
several individuals from different departments of an 
organization. This type of team meets regularly as an off¬ 
line discussion group that seeks to improve quality, 
efficiency, and/or work environment. They have no power to 
reorganize work or change the roles of workers in the 
production process. Suggestions resulting from problem 
solving teams have been found to reduce costs and improve 
product quality, but due to their lack of authority, have 
had little effect on how work is organized or managerial 
behavior. 

b. Special-Purpose Teams 

Special-purpose teams are formed to complete a 
specific duty, such as designing and introducing work 
reforms, meeting with suppliers and customers, and linking 
separate functions. Special purpose teams work on 

decisions at higher levels, creating an atmosphere for 
quality and productivity improvements. 

c. Self—Managing Teams 

Self-managing teams are smaller teams that 

produce an entire product or provide an entire service. 

Team members master multiple tasks and may even rotate 

between jobs within the team. These teams take over all 
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managerial duties, including scheduling, ordering supplies, 
and hiring. Self-managing teams fundamentally change the 
way work is organized, giving employees more control, 
eliminating the need for multiple management tiers, and 
removing bureaucratic barriers. 

2. Task Oriented Classification 

Teams can also be classified by evaluating the tasks 
they are organized to accomplish. These three categories 
are teams that recommend things, teams that make or do 
things, and teams that run things (Katzenbach and Smith, 
1994) . 

a. Teams that Recommend Things 

These teams include task forces, project groups, 
and audit, quality, or safety groups that are asked to 
study and solve particular problems. They usually have a 
set completion date and would benefit greatly from a fast 
and productive start. Teams that recommend things should 
also go to great lengths to gain commitment from the people 
who will implement their recommendations. 

ib. Teams that Make or Do Things 

Teams that make or do things are performance 
driven and should focus on critical delivery points. 
Critical delivery points are the places in the organization 
that most directly influence the cost and value of the 

product. 

c. Teams that Run Things 

Teams that run things are extremely rare. Often 

groups of executives are referred to as a team, but they do 

not meet the fundamental definition of a team. Often, 
because of destructive conflict and a lack of trust, teams 
at the top are difficult to create. Although not as 
efficient as a team, a working group is more easily 


15 



assembled at this level. A working group presents fewer 
risks and allows its members to maintain an individual 
focus. 

C. TEAM LEADERS 

A good team leader has the ability to shape a team's 
vision and provide necessary guidance and oversight. A bad 
team leader can disintegrate a team into a group of 
unfocused individuals that can actually decrease 
productiveness from the start. When forming a team there 
are five critical steps involved (Katzenbach and Smith, 
1994) : 

1. Establish urgency and direction 

3. Set clear rules of behavior, 

4. Set and seize upon performance oriented tasks and 
goals, 

5. Challenge the group regularly with fresh 
information, and 

6. Spend lots of time together. 

Establishing urgency and direction will set the team 
down the correct path and will prevent wasting time trying 
to get the team on track. It is important to pay attention 
to the first meeting and actions, as this will set initial 
impressions. If team members see a hard working, focused 
leader, it is likely that they will be more focused 
themselves. Clear rules of behavior should be set 

regarding attendance, discussions, confidentiality, and 
expected contributions. Establishing a few performance- 
oriented goals will give the team confidence and motivation 
to begin moving forward quickly. Significant challenges 
will unify the team, and when accomplished will provide the 
team with determination and enthusiasm. finally, spending 


16 



time together, both scheduled and unscheduled, will help 
build trust and reliability between team members. 

A good team leader will put team performance first and 
recognize that they need the team's help to accomplish the 
goals. A team leader must delicately balance between 
providing too much command or too little guidance, as both 
can stifle a team's motivation. A team leader should 
follow these five elements for good team leadership 
(Katzenbach and Smith, 1993) : 

• Keep purpose, goals, and approach relevant and 
meaningful 

• Build commitment and confidence 

• Manage relationships with outsiders and remove 
obstacles 

• Create opportunities for team members 

• Do real work. 

Additionally, a team leader should avoid: 

• Blaming people or allowing individuals to fail 

• Excusing away shortfalls in team performance. 

D. CONCLUSION 

"The single most important consideration for those 
responsible for experimentation design is to ensure that 
current expertise is available to support the plan" (Code 
Of Best Practices-Experimentation). 

The potential of a team greatly exceeds the potential 
of the individuals within that team. In an experiment, 
this potential must be realized for success. The Trident 
Warrior '05 team provides an excellent blueprint for 
modeling other experimentation teams. Every individual has 
a particular area of responsibility. In this manner, the 
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team is able to hold itself accountable, and accomplish its 
goal. 



Figure 4 . 


The TW ' 05 Experimentation Team 
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Figure 5. Concept Development 

A. CONCEPT DEVELOPMENT CONFERENCE (CDC) 


The primary event of the concept development phase is 
the Concept Development Conference (CDC). The purpose of 
the CDC is to define the experiment's theme, goals, and 
focus areas. The CDC is an attempt to get the primary 
stakeholders to agree on the overarching concept, or theme, 
for the experiment. Examples of overarching themes include 
the Global War on Terrorism, Expeditionary Strike Group 
Command and Control, and Bandwidth Management. 


B. INPUTS FOR THE CONCEPT DEVELOPMENT CONFERENCE 

Once the team has agreed upon an overarching concept, 
the major stakeholders in the experiment are each given a 
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turn to present possible initiatives for the experiment. 
They also relay relevant ideas, along with thoughts on how 
to improve the experiment. Within the Trident Warrior 
experiments, each stakeholder is given an opportunity to 
briefly define their proposed initiative and link their 
ideas to Sea Power 21, FORCEnet, or other applicable 
military doctrine. By the end of the CDC, everyone should 
have a common understanding of the overarching focus area, 
and the experimentation team can begin to determine which 
initiatives they will investigate. 

When presenting a proposed initiative, the major 
stakeholders should concentrate on these key issues. 

1. Capability Gap 

The fundamental purpose of military experimentation 
should be to provide solutions that support the actual 
warfighter. These solutions may focus on changes to 

current Doctrine, Organization, Technology, Materiel, 
Leadership, Personnel, and/or Facilities (DOTMLPF). In 

order to be considered for military experimentation, 
initiatives should attempt to fill a void, or "gap" in 

current abilities that would allow our forces to achieve 
some desired future or current capabilities. As stated in 
the CDC for TW '06, "We are not doing experimentation for 
experimentation's sake; [our experiments] must fill gaps." 

2. Current Efforts/General Info 

Military experimentation should not attempt to 

recreate knowledge that has already been discovered. 
Initiatives should be researched, and the experimentation 
team should be informed of previous efforts and current 
developments within each field. A general overview of 
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previous/ongoing efforts related to these proposed focus 
areas should be outlined. 

3. Experiment Relevance 

In order to keep the experiment to a manageable 
volume, the CDC will attempt to determine if an initiative 
aligns with the current theme of the experiment. Again, 
teams should have some purpose behind their experiments, 
and not run experiments for the sake of experimentation. 
In military experimentation, this purpose is to fill gaps 
in warfighting capabilities. Allowing too many unrelated 
initiatives will result in the experiment wandering away 
from its overarching concept and becoming too large and 
unmanageable. This wandering is referred to as scope 
creep. According to NETWARCOM's Ron Adamo of FORCEnet 
Experimentation-Future Plans, 

Scope creep is an experiment's most dangerous 
threat. A solid CDC will help by firmly focusing 
the experimentation team (and stakeholders) on 
the key objectives. These objectives can change 
during the rest of the process but only with 
everyone's buy-in. Then possible deviations from 
the original plan can be addressed with a full 
understanding of how they affect the overall plan 
and available resources. 

C. CONCLUSION 

Once the team has agreed on a major focus area and 
relevant initiatives, they should review their roles and 
responsibilities within the new initiatives. This will 
ensure that team members understand the part they will play 
through the rest of the experiment. 

The output of the CDC should be a concentrated list of 
initiatives that enables each of the experiment leads to 
begin concentrating their efforts. If everyone is 


21 



comfortable with this level of focus and direction at the 
end of the CDC then the experiment design team should be 
able to get to work in developing the detailed experimental 
objectives, questions and metrics. 
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Figure 6, 


Technology/TTP Harvest 


A. BROAD AGENCY ANNOUNCEMENT 

In order to attract innovative ideas both from 
industry and from other areas of the military, it may be 
necessary to issue a Broad Agency Announcement (BAA). A 
BAA is a notice from the government that requests 
scientific or research proposals from private firms 
concerning certain areas of interest to the government. The 
submitted proposals may lead to future contracts. 
According to the Federal Acquisition Requirement-Part 35: 

baa's may be used by agencies to fulfill their 
requirements for scientific study and 
experimentation directed toward advancing the 
state-of-the-art or increasing knowledge or 
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understanding rather than focusing on a specific 
system or hardware solution. The BAA technique 
shall only be used when meaningful proposals with 
varying technical/scientific approaches can be 
reasonably anticipated. 

If not using a BAA, some other method of attracting 
outside technologies should be employed. These methods 
should describe the experiment's broadly defined areas of 
interest, the criteria for selecting the proposals, specify 
the period of time during which proposals will be accepted, 
and contain instructions for the preparation and submission 
of proposals. 

BAA's imply program funding is available for an 
eventual buy decision. NETWARCOM does not control such 
funding, so TW does not typically issue a BAA. SPAWAR 
provides technologies in support of FORCEnet, to include TW 
experimentation. Additionally, word of mouth among 
industry, academia, and other military agencies attracts 
sufficient interest from outside sources. 

B. ENTERPRISE DATABASE FOR INNOVATIVE SOLUTIONS 
OBSERVATION NETWORK 

In order to gather beneficial technology/tactics, 
techniques and procedures (TTP) more efficiently, the TW 
experimentation team developed the Enterprise Database for 
Innovative Solutions Observation Network (EDISON), a web- 
based portal through which they can collect and review 
contractor-submitted proposals. EDISON provides a single 
point of entry for all information technology (IT), 
information management (IM), and human systems integration 
(HIS) solutions in support of FORCEnet. Additionally, 
EDISON provides a powerful data-mining portal capable of 
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extracting essential information from existing databases to 
provide the experimentation team with a wide awareness of 
programs and innovative solutions. This single entry point 
for submission of FORCEnet innovative solutions from 
industry, academia, the Department of Defense (DoD), and 
the Navy/Marine Corps team helps to eliminate the costly 
development of redundant capabilities and screen out 
solutions that are not in accordance with the enterprise 
architectural vision. EDISON provides tracking of these 
innovative solutions from discovery to acquisition, 
enabling the selection of optimal ideas from all sources. 
EDISON required that the data only be entered a single time 
and maintains the security of proprietary information 
(EDISON website within EIRE). 

C. TECHNOLOGY DEVELOPMENT CONFERENCE 

After all proposals are collected through EDISON, the 
Technology Development Conference (TDC) is convened to 
determine which proposals will be accepted and incorporated 
into the experiment. The TDC board should be comprised of 
decision makers from the command that is sponsoring the 
experiment (NETWARCOM for TW), as well as representatives 
from the Eleet Sponsor. The first act of the TDC board is 
to throw out all proposals that are not relevant to the 
current experiment, or are not in line with current Navy 
doctrine, organizational structures, or the current 
political environment. The Trident Warrior experiments do 
not seek to evaluate proposals for changes in established 
doctrine at the TDC. These proposals must be routed 
through the Naval Warfare Development Command (NWDC). 
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Proposed doctrine changes can then be forwarded to 

NETWARCOM for experimentation. 

The TDC board should also remove any proposals that 
are not technically feasible or will not fit into the 
agreed upon timeline for the experiment. The Trident 
Warrior experiments have found it useful to divide the rest 
of the TDC into 2 phases. During the first phase, 
interested government entities are permitted an opportunity 
to give a brief presentation on technologies they wish to 
include in the experiment. Other government entities may 
want to include their technology for the purpose of testing 
or demonstrating capabilities. During the second phase 
contractors are permitted to give a brief presentation on 
technology they feel would be beneficial to include in the 
experimentation campaign. During this phase, the only 
parties present are the TDC board and the contractors 
giving the presentation. This separation is intended to 
protect contractor's proprietary information. 

In order to be approved by the Technology Development 
Conference, a proposal must: (1) be a FORCEnet innovative 

solution adding value to the warfighter by enhancing 
current capabilities without detracting from current 
readiness or capabilities; (2) be aligned within the 
defined high level FORCEnet capabilities requirements, and 
concepts; (3) be consistent with, and build up, fleet 
priorities; and (4) provide feedback to the 

requirements/acquisition process for Navy wide delivery of 
FORCEnet capability. Additionally, within Trident Warrior, 
The objective is usually to leave the technology onboard 
the ships provided by the fleet sponsor, so the proposal 
should attempt to deliver end-to-end, supportable and 
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deployable leave behind capabilities. The amount of 
funding required from the experiment's sponsoring command 
(NETWARCOM for TW) would factor into the decision as well. 
(EDISON website within EIRE). 

D. CONCLUSION 

Technology and accepted tactics, techniques and 
procedures (TTP) should be gathered in an effort to 
optimize the experimentation effort. The level that these 
gathered technologies play within the experiment is 
dependent on the experiment's overarching focus area and 
the initiatives of the experiment. Regardless, the 

gathering of technology and TTP is a crucial step in the 
experimentation process. This step will ensure that the 
experimentation team fully understands the technologies, 
systems, tactics, techniques and procedures (TTP) they are 
about to test, and that the team can present new and 
innovative methods of employing these technologies and 
systems. 
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Figure 7 


A. ASSET IDENTIFICATION 

It is very important to get the entire team to accept 
the experimentation campaign from the very beginning. An 
established commitment from every team member will promote 
trust and confidence within the team. Military 

experimentation has an added challenge, in that the 
platforms and some of the assets required to perform the 
experiment do not belong to the experimentation team. 
Military equipment and platforms are produced to support 
and defend the Constitution of the United States in the 
most effective manner possible. Sometimes this is means 
fighting wars, and sometimes this means enhancing 
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warfighter capabilities through experimentation. Due to 
the volatility of world affairs, it is often difficult to 
get a solid commitment from the military assets needed to 
perform the experiments. To make the situation worse, 
military experimentation often requires that experimental 
systems be installed on military platforms, requiring an 
even greater commitment. 

1. Require a Commitment 

A firm commitment must be obtained from the commanders 
of all required military assets. Without the confidence 
that all committed assets will be available at the time of 
experimentation, planners cannot begin to develop the 
experiment's objectives. Installation of experimental 

systems should be scheduled for the committed systems and 
platforms. Scheduled installation could take as long as 
two calendar years to complete. The commitment required 
from these commanders is very burdensome, but also very 
necessary. 

2. Get Buy-In 

The experimentation team should remember who has the 
ultimate authority over the assets. The most important 
support to acquire is the individual with the most 
influence over the assets. If you have the support of the 
boss, the staff will follow. In order to obtain this 
support, the experimentation team should be prepared to 
defend their efforts and detail the purpose of their 
campaign. 

3. Be Flexible 

When circumstances begin to change, it is important 
that the experimentation team remain flexible. World 
events frequently dictate the redeployment of assets. 
Platforms and systems that were reserved to carry out the 
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experiment may no longer be available. New resources may 
be available in their place, or a reduced quantity may have 
to suffice. It is important to maintain a flexible 
timeline and an adaptable experimentation plan in order to 
overcome these last minute changes. An experiment that 
improves knowledge is valuable, even if the knowledge 
gained is less than expected. 

B. DEVELOP EXPERIMENT OBJECTIVES 

Within this stage of the Trident Warrior process, the 
experimentation team attempts to refine the major 
initiatives agreed upon in the Concept Development 
Conference. At the CDC, the team agrees upon an 
overarching theme for the experiment, as well as some high 
level initiatives or focus areas. The team then splits 
these focus areas into specific objectives to accomplish, 
and formulates questions within each objective. Objectives 
relate to a specific capability that is to be explored 
through experimentation. 

Developing objectives is fundamentally a social 
process of forming a shared understanding in order to 
determine what is to be addressed by the assessment. 
During a C2 assessment problem formulation, the analytic 
problem is decomposed into appropriate dimensions such as: 
functions, mission areas, structures, command echelons, and 
C2 systems. 

Problem formulation is an iterative process that 
evolves over the course of the study. [This] 
phase should identify the context of the study 
(i.e. actors; threats; relevant previous studies; 
and political, social, historical, economic, 
geographic, technological environments) and 
aspects of the problem-related issues (i.e. 
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issues to be addressed, assumptions, independent 
variables, and constraints) ( COPB for C2 
Assessment , 54) . 

TW objectives are broken down into exceptional detail 
by decomposing each into the following categories for each 
initiative: 

• Initiative Statement (a high-level description of the 
initiative's purpose) 

• Objective (operational or technical capability to be 
produced 

• Military Utility Assessment (MUA) Recommendation 
(recommendation that will be made to Military Utility 
Assessment Board if the objective is a success. 

• Questions/Proposal (Specific aspect of the initiative 
to be addressed. 

• Operational conditions required to produce valid data 
relevant to the question being asked 

• Systems conditions required 

• Information conditions required 

• Attributes and measures that will be evaluated 

• The data required to produce the assessment, which 
will produce the measures. 

This step can take several months to complete because 
a typical TW can generate up to 150 separate initiatives. 
But when these questions are focused at the right level of 
detail, the rest of the event design is expedited. 

A formal statement of the objective can be used to 
make the experiment more productive. "The statement of the 
[objective] should be formulated so as to specify the whole 
issue under study, not just the specific hypothesis under 
detailed analysis" ( COBP-Experimentation , 129) . This high 
level description will help the team identify all the 
elements within the objective and ensure that proper 
controls are in place. The COBP for Experimentation 
suggests that when writing an objective statement, it is 
important to ensure that, regardless of the result of the 
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experiment, the overall knowledge of the event is 
increased. This would require that the statement be 
written in such a way that it compares two outcomes. This 
typically requires the objective to establish a base line, 
or placebo, with which to compare the test case. 
Unfortunately real world experimentation is usually unable 
to create two identical situations in which to compare test 
and baseline results. The Trident Warrior experiments 
consider current systems, technologies, TTP, etc. as the 
baseline for experimentation. 

Once the objectives have been outlined, the team can 
begin to develop the specific questions to be asked within 
each of those objectives. These questions will lead to 
attributes that need to be determined from the outcome of 
the experiment. Attributes will dictate what measures 
(data) should be collected during experimentation. The 
output from this step should provide the team with the 
required conditions, events, and data that must be 
collected to produce the required measures. 

1. An Ideal Experiment 

Although running an ideal experiment is impossible, 
many factors can be manipulated to make conditions as ideal 
as possible. An ideal experiment will control all sources 
of variation except the variable of interest. In order to 
do this, the objectives will require a theory that 
identifies limiting conditions, as well as the casual 
factors that influence the parameter/quantity of interest. 
Additionally, an ideal experimentation will measure all 
active variables and the limiting conditions precisely, 
correctly, and reliably. 
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According to the Code of Best Practice for 
Experimentation, an ideal experiment: 

• Manipulates only one independent variable at a time, 

• Observes change in only one dependent variable, 

• Excludes, or controls for all relevant extraneous or 
intervening variables, 

• Involves valid, reliable, precise, and credible 
measurement of all variables, 

• Includes enough data collection opportunities to 
support the inferences needed to translate findings 
into actionable knowledge, and 

• Generates findings, interpretations, and insights. 

In an ideal experiment, we are trying to manipulate a 
single variable and measure its impact on another single 
variable in order to draw a cause-effect relationship. 
Again, in real world situations ideal conditions are 
sometimes not feasible. Real world experimentation 

attempts to run a larger quantity of experiments in order 
to observe and account for the influence of outside 
factors. If the experimentation team is not cautious, 
variables are inadvertently manipulated, outside variables 
cannot be controlled, or the desired variable is not 
effectively measured. These instances can change an 
experiment's expected outcome, and possibly detract from 
the general knowledge base on the subject matter. 

Bad experiments, which cloak weak or false 
knowledge in an aura of science, will make 
mischief, not music. At a minimum, they will 
slow the process of understanding the impact of 
innovation on transformation (by forcing other 
research and experimentation to learn and 
demonstrate that they are wrong) and cost money 
(by encouraging investments that will not pay off 
as expected) . At a maximum, bad experiments well 
lead to flawed mission capability packages that 
fail in the field. (COBP-Experimentation, 140) 
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C. CONCLUSION 

Once the experimentation team has defined the 
experiment's objectives, they can then begin to formulate 
specific questions. These questions have matching 
attributes that will provide linkage between the questions 
within the initiative and capabilities in the FORCEnet 
Concept or tasks in joint documents. The experimentation 
team will then determine what data should be gathered to 
validate these attributes and answer the questions. 

Additionally, each objective will need to specify the 
overall conditions under which this data will be gathered. 
In the following step, the initiatives will be diagramed 
and checked to ensure that all of these components are 
complete and correct. 
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Figure 8. Process Diagrams 


Modeling the individual objectives of an experiment 
will allow the team to analyze the various relationships 
within each focus area. Models highlight any inputs, 
variables, and processes when developing the experiment 
objectives. 

Trident Warrior requires a high level IDEFO model be 

developed for each objective that shows how the objective 

contributes to that initiative. For the objective level 

and below, more targeted models are to be employed, 

depending on the analytic intent of the objective. 

Multiple model types can be included within the different 

areas of research to maximize their effectiveness. The 

analysis team, modeling assist team and initiative leads 
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will determine which models best fit into each situation, 
and specify what models need to be constructed. Modeling 
is an important step in comprehending the experiment and 
its objectives and evaluating each objective for 
thoroughness. 

A. QUAD CHARTS 

In order to uniformly and effectively display the many 
objectives within each experimentation campaign, the 
Trident Warrior team has developed its own design diagram, 
called a Quad Chart. A Quad Chart should be organized for 
each objective within the experiment, and will "display all 
pertinent information about an objective's purpose and 
requirements, provide an easy to examine display, in a 
format that can be used for presentations, and provide 
easily visualized information that can be edited and 
corrected before more detailed planning and FIRE input" 
(TW-05 Short Planning Directions, Appendix A) . A Quad 
Chart is useful because it is a method of describing an 
entire objective in a single diagram. 

A header should be above the Quad Chart, containing a 
one sentence initiative statement and the point of contact 
for that initiative. The first quadrant should define the 
objective with a single, over-arching statement of the 
objective's goal of what is to be learned or accomplished. 
The second quadrant should define the recommendations that 
will be presented to the Military Utility Assessment (MUA) 
Board if the objective is successful. Together, the 
objective statement and the MUA recommendation describe 
what is to be accomplished within this objective. The 
third quadrant should contain the questions or proposals 
that would support the objective and MUA recommendation. 


38 



Multiple questions may be required per objective. The 
collective set of answers to these questions will provide 
the information to be produced for this objective and its 
MUA board. 

Quad 
Page-1 


One sentence Initiati^'e statement Point of Contact 

ObjectiAe 

Single OA'erarchmg statement. 

MUA Recomendation 

PrecieA ed benefit if objecth e is 
successful 

Quesrions/Proposal 

Multiple questions, if needed, each 
supporting an Information Goal. 

Operational Conditions 
Statement of all required conditions. 


Same one sentence InitiatiA e statement 

Point of Contact 

System Conditions 

Information Conditions 

Statement of all required conditions. 

Statement of all required conditions. 

Attributes 

Data Sources 

List of all Measures needed to 

List of all rtpes of Data Sources 

answer Questions, mcluding details. 

needed to determine Measures. 


Figure 9. Quad Chart 

Quadrants 4 through 6 will gather general conditions 
to frame the situations and collect the correct 
information. The fourth quadrant will contain a statement 
of all the required operational conditions, specifying 
which people or organizations will be responsible for 
conducting the operations within the objective. The fifth 
quadrant will contain a statement of all the required 
system conditions for this objective. The sixth quadrant 
will contain a statement of all the required information 
conditions for this objective, covering information type, 
load, protocols, distribution, etc. The seventh quadrant 
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will provide a general description of the attributes 
required to answer the questions or meet the proposals in 
quadrant 3. Finally, quadrant 8 will list all the data 
sources needed to determine the measures in quadrant 7. 
This will include whether the data is subjective or 
quantitative, as well as whether this data will be recorded 
electronically, through chat logs, in observation logs, or 
through surveys. 

B. IDEF MODELS 

IDEF or Integrated DEFinition language is the 
modeling technique used to model functional 
processes and corresponding information to 
support Functional Process Improvement. The 
Department of Defense selected the IDEF [model] 
as the technique for functional managers to use 
for function (activity) and information (data) 
modeling. IDEFO was selected as the standard 
technique for activity modeling. (McConnelly) 

IDEFO is used to model the decisions, actions, and 
activities of an organization or system, in order to 
communicate the functional perspective of a system. IDEFO 
models describe the functions that are performed as well as 
what is needed to perform these functions. The basic 
structure used in IDEFO modeling is the referred to as the 
ICOM format. ICOM stands for Input, Control, Output, and 
Mechanism. The center square of the ICOM represents the 
activity, or the function to be accomplished. Inputs are 
all the things that will be transformed by the activity, 
including materials and information. Controls are elements 
that govern or constrain the activity, including budgets 
and policy. Output is the result of the activity. 

Mechanisms are the elements that take part in or support 
the activity, to include people, systems, and facilities. 
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Constraint 



Figure 10. Generic ICOM for an IDEFO model 
Constructing a Context Diagram is the most basic 
element of the IDEFO function modeling method and will 
represent the function at its highest level, limiting the 

scope of the model. A functional decomposition of the 

Context Diagram will produce the major activities required 
to complete the activity described in the Context Diagram. 
IDEFO modeling is an iterative process and should be 
continued until reaching the level of detail that is 
desired. Subsequent models can continue to be broken down, 
with the new models in each level explaining actions 

required to complete the actions of the previous level. 
This process will continue until the IDEFO model has been 
decomposed into its simplest state. Based upon this 

decomposition, it will be possible to determine if any 
inputs, controls, outputs, mechanisms, or relationships 
within the experiment were overlooked. Then it may be 

necessary to update the context diagram to include these 
changes. 
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Figure 11. A Context Diagram for Bandwidth Management 

Objective (from: TW '05) 


C. OPERATIONAL SEQUENCE DIAGRAMS 

The functional decomposition of IDEF models can 
produce a complex output with multiple layers of 
information. These layers of information can sometimes be 
confusing and difficult to follow. Operational Sequence 
Diagrams (OSD) provide a single page, flow chart view of 
relationships between organizations, people, and/or 
technologies involved in a task. An OSD is essentially a 
flow chart that has been applied to a functional process to 
model the sequence of data or information as it flows 
through the system. An OSD can then be used as a planning 
tool or an analysis tool when developing an experiment. 
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Figure 12. OSD Model for Bandwidth Management Objective 

(From: TW '05) 


D. CONCLUSION 

Without these diagrams, it is possible to overlook 
relationships that may factor into the scenario. By 
drawing process diagrams for each objective, the 
experimentation team hopes to ensure that each objective is 
complete and correct. This includes the questions 
involving the desired attributes of each objective, the 
data needed to answer these questions, and the overall 
conditions that this data will need to be collected under. 
The following steps will designate specific scenarios to 
set the conditions for each objective and set up a 
comprehensive schedule for actually running the experiment. 
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VII. EXPERIMENT DESIGN/EVENT DEFINITION 


Phase 

1 

2 
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Figure 


Experiment Design/Event Definition 


A. EXPERIMENT DESIGN 

The product of experimental objectives development 
provides the team with the overarching focus area, the 
major initiatives within the focus area, and the objectives 
that support those initiatives. The objectives will now be 
aligned with specific questions that identify the 
attributes to be gathered by the experiment. An attribute 
is a general characteristic that can apply to any number of 
tasks, systems or processes. This enables attributes to 
provide the link between the objectives. An attribute is 
associated with a certain capability, which is expressed as 
a single word (i.e. timely, flexible, redundant) . This 
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characteristic can be desired or undesired. In order to 
determine the attribute for an objective, specific measures 
must be taken. These measures comprise the data that is 
collected. Thus, data is collected to determine the 
measures and answer the questions, which describe the 
objectives that comprise the initiatives, making up the 
overarching concept of the experimentation campaign. 

Anatomy of an Experiment 



Figure 14. Breakdown of an Experiment 
For example, if one focus area of an experiment is 
defined as mission planning, a possible experimental 
objective could be to develop the tools to support 
planning. The experimentation team may prepare a number of 

questions or goals that need to be achieved in order to 
satisfy this objective. A list of possible goals/questions 
for this example could be: (1) Provide rapid reach-back, 
(2) Provide Complete Information, (3) Provide a consistent 
Common Operating Picture (COP), and (4) Provide easy 
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collaboration. When these goals/questions are stated, it 
becomes easy to determine the attributes required to meet 
each goal and satisfy the objective. The attributes for 
this example would be: (1) rapid, (2) complete, (3) 

consistent, and (4) easy. The first three objectives are 
not difficult to comprehend, and only the fourth goal would 
require further explanation. The measures required to 
validate these attributes can now be determined. It is 
easy to become distracted with the naming of attributes, 
while the real focus should be on the measures. In order 
to validate the first attribute (rapid), it might be 
necessary to measure the quantity of time required to 
receive information from a system. This measurement should 
be defined as the interval of time between submitting a 
request for information and the receipt of that 
information. Measurements can also be subjective, but it 
is usually more convenient to put this subjective 
observation on a quantitative scale (i.e. on a scale of 1- 
5, how comfortable were you with the subject matter, with 1 
being not comfortable, and 5 being very comfortable). This 
method allows the experiment to collect subjective data, 
and the analysis team to perform a quantitative analysis. 
This subjective measurement, or the measurement of the time 
delay in the previous example, is the actual data 
discovered by the experiment. 

Once the details of the objectives have been 
established, the experimentation team can now begin to 
combine these objectives into a single experiment with 
unique scenarios that will produce the desired operational 
conditions, systems conditions, and information conditions. 
The Code of Best Practices for C2 Assessment defines a 
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scenario as "A description of the area, the environment, 
means, objectives, and events related to a conflict or a 
crisis during a specified time frame suited for 
satisfactory study objectives and the problem analysis 
directives." A scenario should represent a plausible real 
world situation that would reflect the factors that are 
needed for each objective. Scenarios may also require 
smaller vignettes in order to produce all of the required 
conditions for each objective. "A vignette is primarily 
used for smaller scenarios, particularly as excursions from 
the main scenario" ( C0BP-C2 Assessment , 167) . The 

scenarios and vignettes set the conditions and restrictions 
for each objective. These conditions allow comprehensive 
analysis and create a structure where the results of the 
analysis can be understood and interpreted. "The purpose 
of scenarios is to ensure that the analysis is informed by 
the appropriate range of opportunities to observe the 
relevant variables and their relationships" ( COBP- 
Experimentation , 200) . Scenarios must be selected, crafted 
and adapted to ensure that they support the objectives of 
the experiment. 

"The use of a single scenario, even one that has been 
carefully crafted to focus on the key issues of an 
experiment, invites suboptimization and narrows the range 
of applicability for the findings" ( COBP-Experimentation , 
221 ) . 

B. EVENT DEFINITION 

1. Measures of Effectiveness 

In its simplest form, military experimentation tries 
to evaluate operational capabilities. In order to perform 
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these evaluations, previous experimentation teams have used 
descriptions of Measures of Effectiveness (MOEs) and 
Measures of Performance (MOPs). The Defense Acquisition 
University defines a Measure of Effectiveness (MOE) as "a 
measure of operational success that must be closely related 
to the objective of the mission or operation being 
evaluated," and Measures of Performance as "measures of a 
system's technical performance expressed as a distinctly 
quantifiable performance feature (i.e. speed, payload, 
range, time, or frequency) 

When attempting to relate MOEs and MOPs to specific 
items within the experimentation process, these definitions 
can be confusing. MOEs are always a higher level of 
measurement than MOPs, but exactly what they are describing 
depends on the context. Within our previous example of 
developing the focus area of mission planning with an 
experimental objective of developing tools to support 
planning, there are multiple ways of naming MOEs and MOPs. 
The initiative of mission planning could provide a Measure 
of Effectiveness that would describe the overall quality of 
the mission plan. Under this MOE, a possible MOP would be 
provided by the objective in a measurement of the quality 
of the planning support tool set. However, other potential 
MOEs are available. The attributes within each goal could 
provide a Measure of Performance. Within this example 
those measures would be rapidity, completeness, 
consistency, and ease. Similar to the attributes these 
measures were developed from, the fourth MOP would be the 
most difficult to comprehend. 

Another option for the Measure of Effectiveness (MOE) 
would be the experimental objective. Listed in the last 


49 



example as a MOP, this measure could describe the quality 
of the planning support tool set. The MOPs under this MOE 
would be the measures of rapidity, completeness, 
consistency, and ease, as described by the attributes. 
Regardless of which option is chosen, the nomenclature must 
remain constant through the experiment. If MOEs are 
determined at the initiative level, it must remain that way 
across the experiment. MOPs abide by the same restriction, 
if MOPs are described by the attributes, it should be 
constant across the experiment. 

The actual level within the experiment that the 
MOPs/MOEs define is context dependant. The Modular Command 
and Control Evaluation Structure (MCES) mandates that there 
be rigid boundaries between MOEs and MOPs, but it does not 
say where to establish those boundaries. 


Environment 



DP: Dimensional parameter 
MOP: Measure of C2 
system performance 

MOE: Measure of 02 
system effectiveness 

MOPE: Measure of force 
effectiveness 


Eigure 15. Relationships Among Classes of MOMs (from: MCES) 
The terms MOE and MOP were developed to provide 
insight on the experiment, but they may actually cause more 
confusion than insight. It is possible to describe the 
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experiment using only initiatives, objectives, attributes 
and measures. 

2. Master Sequence of Events List 

Once a list of desired objectives with corresponding 
scenarios is produced, along with a list of desired 
measures to be collected, the experimentation team will 
begin to layout a schedule of events for the actual 
experiment. This schedule of events is called a Master 
Sequence of Events List (MESL) . The MESL will combine the 
experimentation campaign's multiple scenarios together with 
the schedules for the participating assets, to produce a 
document that delineates a time and flow for all of the 
major activities within the actual conducting of the 
experiment. 

The MESL must first consider the schedule of the 

assets that will be participating in the experiment. 
Within Naval experimentation, this schedule will include 
time in port, time spent in transit, and time on station. 
Additionally, this schedule will include time for specific 
events such as flight operations and crew watch-standing 
rotations. The experimentation team will have limited 
control over these aspects of the schedule. 

Next, the MESL should consider the various scenarios 

required to complete the experiment. Occasionally, 

multiple scenarios can be run at the same time, but great 
caution should be taken to ensure that they do not 

conflict. The various scenarios should be scheduled in an 
orderly fashion, and should not impose undue levels of 
stress upon the ship and its crew. The conditions required 
to answer the each question need to be present at least 
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once within the running of the experiment. Ideally there 
would be multiple opportunities to complete each objective. 
Scenarios that correspond to higher priority objectives 
should be run earlier in the schedule, permitting them to 
be bumped down in the case a conflict. If higher priority 
scenarios are not run when scheduled and no excess time is 
available, a decision will have to be made on which lower 
priority objectives/scenarios can be omitted. The 
objective of the MSEL is to answer all of the questions for 
each objective within the given timeframe. 

C. CONCLUSION 

Once the overall plan for the experimentation campaign 
has been laid out, the team will begin to grasp the volume 
of the task they are undertaking. It is important that 
every scenario has been laid out in specific detail. 
Concentrating on one detail at a time and not becoming 
overwhelmed by the sheer volume will see the experiment to 
completion. Once the scenarios are defined and developed 
in detail, the experimentation team can begin to 
concentrate on how and where the individual pieces of data 
will be collected. 
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VIII. 


DATA COLLECTION PLAN 
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Figure 16. Data Collection Plan 


A veteran experimenter will require only a quick 
glance at the Trident Warrior Experimentation Process to 
realize that no specific step is included to analyze data. 
This is a very dangerous omission, as it does not express 
the significance of the Data Analysis Plan. Within Trident 
Warrior, the Data Analysis Plan (DAP) is written in 
conjunction with the Data Collection Plan (DCP), creating a 
Data Collection and Analysis Plan (DCAP) . This is done 
immediately following the Mid-Planning Conference (MPC), 
approximately 4 months before the actual experiment. 
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A. DATA ANALYSIS PLAN 

What makes Trident Warrior different from other 
naval assessment events is the level of detail of 
the analysis [planning]. That level of detail 
can be attributed to the Trident Warrior Process. 

The strict compliance to this process is what 
ensures event consistency and allows us to 
maintain a high standard in our FORCEnet 
assessments. CAPT Chris Abbott, director of 
FORCEnet Innovation and Experimentation. 

(Poeltler and Gallup, 29) 

While it may seem obvious that data must be collected 
before it can be analyzed, it is not so clear that you must 
know how the data will be analyzed before you can know what 
data to collect. By preparing the data analysis plan 

before, or in conjunction with, the data collection plan, 
the experimentation team ensures that only relevant, useful 
data is collected. "In economic terms, the data analysis 

plan acts as the source of demand. The data collection 

plan acts as the source of supply" (NATO COBP for 
Experimentation, 224) . Without the data analysis plan, it 
becomes very tempting to simply collect the data that is 
easiest to access, rather than the data that is most 
influential to the experiment. In order to provide the 
necessary data, the data analysis plan must be linked to 
the data collection plan, the post-experiment modeling 
plan, and the material necessary to revisit, revise, and 
perhaps extend the conceptual model underlying the 
experiment. This ensures that the data collected will be 
directly applicable to the overall purpose of the 
experiment. 

The first step of data analysis is "to identify the 

independent variables and their measures, then the active 
variables and their measures, and finally the intervening 
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variables (including those that are to be controlled in the 
experiment) and their measures" (NATO COBP for 
Experimentation, 226) . A quick glance will also show that 
much of the analysis is very in-depth, and it is often a 
requirement to have a statistical expert within the 
experimentation team. Analysis can be categorized into 
three general phases: descriptive analysis of individual 
variables, bivariate analysis of relationships, and 
multivariate analyses of larger patterns. 

1. Individual Variables 

Once the data has been converted into the form 
required for analysis and formulated into an integrated 
data set, the first analytic effort should be a descriptive 
analysis of each variable of interest. "These descriptive 
analyses are performed to (a) identify and correct data 
anomalies, (b) understand the distribution of each 
variable, and (c) identify any transformations of those 
variables with distributions that may make analysis 
misleading" ( NATO COBP for Experimentation , 227). When 
identifying and correcting data anomalies, analysts should 
search for invalid or illogical data. The Data Analysis 
Plan should include time to research these anomalies in the 
original records and correct them. Uncorrectable data will 
need to be excluded from the data set. Next analysts 
should review the distribution of each variable, searching 
for variables that would cause an error within the 
analysis. Typical examples include variables that have a 
single value or variables that in only a small number of 
cases differ from a single value. These variables should 
also be removed because they will not be able to contribute 
to the differences in the variables of interest. 


Additionally, in this step analysts should search the data 
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for outliers, or small groups of data that are very distant 
from the bulk of the data. Outliers will make the 
application of simple linear statistics invalid unless they 
can be transformed (possibly using a log function), 
excluded using the creation of a new variable, or 
subdivided from the bulk data using a dummy variable. The 
last technique actually suggests that the data is from a 
bimodal distribution. 

2. Bivariate Analyses of Relationships 

Bivariate relationships are typically considered to be 
little more than a building block for multivariate 
relationships. However, because most hypotheses are 

started in bivariate terms (IF A, THEN B, under CONDITION 
C), bivariate relationships comprise a majority of testable 
propositions. When attempting to identify relationships 
between various factors and variables, the order for 
conducting bivariate analyses is (NATO COBP for 
Experimentation): 

1. Dependent variables; 

2. Control factors 

3. Control factors and dependent variables 

4. Independent variables 

5. Control factors and independent variables 

6. Independent variables and dependent variables 

Dependent variables are tested first in order to 
ensure that they are not correlated. If two dependent 
variables are strongly correlated, only one of them should 
be fully analyzed. This also means that the underlying 
conceptual model should be revisited because it incorrectly 


distinguished 

between 

two 

factors 

that 

were 

closely 

associated. 

Variables 

with 

modest 

levels 

of 

bivariate 

correlation 

should be 

kept 

in the 

model 

and 

analyzed 
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separately, as they could prove important in building a 
comprehensive model. 

If a strong relationship between control factors is 
found, one of them should be removed to save efforts in 
analyzing similar relationships. 

By examining the relationships between control factors 
and dependent variables, analysts can quantify the 
effectiveness of the experiment. In a well-designed 
experiment, control factors have no direct impact on the 
data from the experiment. A major problem in most human 
subject experiments is learning through repetition, and an 
experiment should be designed to limit this. 

Next the analyst should check for correlation between 
the individual independent variables. Most analytic tools 
assume that the independent variables are unrelated, and 
thus, having correlated independent variables will yield 
false statistical explanations. Correlated independent 
variables will be handled in the same manner as previous 
variables: they can be combined, they can be analyzed 

separately, or one of them can be thrown out. 

Similar to the relationship between control factors 
and dependent variable, the relationship between control 
factors and independent variables could indicate a flaw in 
the experimental design or in the conceptual model. A 
correlation between control factors and dependent variables 
does not necessarily contaminate the data, but this 
correlation should be considered when conducting the 
multivariate analyses. 

Finally, the purpose of the experiment is to examine 
the relationship of the independent and the dependent 
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variables. These tests of correlation provide direct 
answers to the hypothesis posed in a conceptual model. 

3. Multivariate Analyses 

The multivariate analyses are the end effort to 
examine the data in the full affects of the surrounding 
environment. After running all of the individual bivariate 
analyses, the experimentation team has ensured that the 
effects on the dependent variables come as a direct result 
of a change in the manipulated variable. Ideally, this 
will allow the experimenters to draw a cause-effect 
relationship between the two variables. 

B. DATA COLLECTION PLAN 

"The data collection plan includes all the variables 
to be collected, all the places where they are collected, 
all the means of collection, and all the places the data 
will be stored for processing" ( COBP for Experimentation , 
241) . The data collection plan is required to be 

simultaneously very broad and also very specific. It must 
not only specify who, when where, and how all data is to be 
collected, but it must also cover all training and support 
required, proficiency standards and quality control. A 
plan for the archiving and processing of raw data into a 

form that can be processed by the data analysis plan must 
also be included. The key steps of a data collection plan 
include ( COBP for Experimentation , 242): 

• Specifying the variables to be collected, 

• Identifying the collection mechanism for each 
variable, 

• Ensuring access for collecting each variable, 

• Specifying the number of observations needed for 
each variable and checking to ensure they are 
expected to be generated. 
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• Identifying the training required to ensure 
quality data collection, 

• Specifying the mechanisms to ensure data capture 
and archiving, and 

• Defining the processes needed for data reduction 
and assembly. 

In addition to this sequence, there are also multiple 
ways of collecting the data: automated collection, 

recording for later reduction, surveys, subject testing, 
and human observation. 

1. Automated Collection 

As experiments, particularly in the Command and 
Control functions, become increasingly automated, it 
becomes more convenient to automate the data collection as 
well. Typical automated collection focuses on system 

loads; workstation loads; system, workstation, and 
application usage; as well as a comparison between test 
subject's perceptions and the "ground truth." The data 
collection plan must specify where and when this data will 
be collected, how the data can be collected without 
affecting the experimentation scenario, how to synchronize 
the data collection and the experiment, and where the data 
will be archived. 

2. Recording for Later Reduction 

Due to the limited supply and abilities of human 
observers, experimentation teams often rely on audio or 
video recordings to supplement their data collection 
process. Unfortunately, even recordings are not always a 
thorough means of capturing all of the events within an 
experiment. There is the possibility of catching one side 
of a conversation, the test subject stepping out of camera 
or microphone range, and any number of technical problems 
with the recording equipment. Unless the experimenters are 
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properly prepared for the recording process, the material 
they record will likely be incomprehensible, inadequate, or 
incomplete. 

3. Surveys 

Surveys are popular data collection tools because they 


are easy to 

review 

and 

can 

be used in multiple ways. 

Surveys can 

be used 

to 

gain 

knowledge 

about 

potential 

subjects. 

Subjects 

can 

be 

tested on 

their 

expertise. 


familiarity with the equipment, and their familiarity with 
other subjects. Subjects can also be surveyed during the 
experiment to determine their perceptions, knowledge, 
ideas, understanding, attitudes, insights, and situational 
awareness as they are being tested. The experimentation 
team can also be surveyed during the experiment to gather 
their insight on how well the systems are performing and 
how they can be improved. 

Surveys can pose problems within an experiment. A 
survey can lock the experimentation team into only 
observing the predetermined items that were mentioned in 
the survey. To combat this problem, it is good to include 
some open-ended questions in your survey that ask the 
subject for additional reactions and observations. Another 
potential problem is that surveys required time. Subjects 
will require additional time to complete surveys, and this 
time must be scheduled into the master sequence of events 
list. The only way to determine how much time is necessary 
to complete the survey would be to pretest the survey using 
similar respondents. Using the experimentation team to 
pre-test the survey will not provide adequate time for the 
less familiar test subjects to complete the survey. 
Precise attention must be paid to the wording of surveys. 
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Misunderstood or confusing questions will not provide the 
data desired. If a survey is conducted within multiple 
phases of an experiment, it will tend to impact future 
behaviors. In order to prevent the data from reflecting 
this "learning curve" by the subjects, it is sometimes 
beneficial to show the subjects the survey before the first 
phase of the experiment. The experimentation team should 
also realize that a poorly structured survey creates 
unnecessary work on the data analysis team. Multiple 
choice or scaled questions are easier to process 
analytically. Surveys can be a very effective means of 
gathering data that would be otherwise unobservable, but 
due to the many risks, any experimentation team using 
surveys should include a team member with expertise in 
survey design and procedures. 

4. Subject Testing 

Subject tests are generally used to gather information 
on proficiency or personal characteristics. "Proficiency 
tests are used when the results of the experiment are 
expected to depend, at least in part, on the particular 
skills of the individuals or teams participating. If 
proficiency is a control or is being used to assign 
subjects in order to control it out of the experiment, 
testing will be needed" ( COBP for Experimentation , 249) . 
In this situation, it is beneficial to not release the 
scores of the proficiency test to the subjects until after 
the experiment. 

Subject tests are also used to test for personal 
characteristics. Typical characteristics tests, such as IQ 
or personality tests, should be chosen from tests that have 
already been published and validated. There is no reason 
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for the experimentation team to attempt to reinvent these 
tests and possibly give the experiment suspect credibility. 
These tests should be administered professionally. Data 
from these tests should not be linked with the individual 
subjects through processing. 

5. Human Observation 

a. Observer Selection 

Observers are a key part of the data collection 
process and should have an appropriate background with 
respect to the experiment's focus. Observers should have 
some proficiency at observation and additional training 
should be available. Within Trident Warrior, observers are 
typically underway on vessels, and it is important that 
they blend in with their environment and do not cause an 
unnecessary distraction. Observers must be proficient in 
knowing what type of data they need to collect and how to 
collect it. It is not uncommon for an observer's shift to 
exceed 8 hours, and therefore, observers should be 
physically fit and capable of handling the stress of their 
job. 

b. Observer Scheduling 

Observer scheduling and positioning is a key 
issue. Time should be scheduled for additional training 
and familiarization. It is good to incorporate additional 
time into the schedule for observers to review their notes 
and format the data for analysis. It is also a good idea 
to have extra observers available. When they are not 
observing, they can be used in a quality control function. 
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C. CONCLUSION 

Typically, the DCAP should be published well before 
the beginning of the experiment, in order to provide the 
experimentation team and observers time to familiarize 
themselves with the plan. However, within Trident Warrior, 
access to FIRE permits everyone to view the individual 
elements that will make up the DCAP, allowing the team to 
shorten the timeline and make changes to the DCAP right up 
until the last minute. 
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IX. EXECUTION 
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Figure 17. Exec 

A. BEFORE THE EXPERIMENT 


1. Data Collection 

Whenever possible, the experimentation team should 
collect any available, pertinent data before the experiment 
begins. This is typically limited to background and 
expertise of the test subjects. Determining differences 
between subjects can ease data analysis and lend 
credibility to the experiment. It is important to archive 
this data immediately, and not become distracted by the 
upcoming experiment. If the data is not archived before 
the experiment starts, there is a good chance that it will 
become lost in the shuffle. Data should always be saved in 
both raw and processed forms. Raw data will permit the 
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analysis team to review the data, recreate the scenario, 
and even fix corrupted or incorrectly processed data. 
Whenever dealing with personal data, it is important to 
ensure that all personal information will be kept private. 

Privacy protection is an important element that 
cannot be overlooked. It is imperative that the 
subjects understand that they are not being 
evaluated (the systems, doctrines, organizational 
innovations, under study are being assessed, not 
the subjects) and that no links will be made from 
individual participants to scores and/or results. 

Not only must the understand this, but they must 
also be confident that this is true. Failure to 
instill this confidence can result in distorted 
results. (COBP for Experimentation, 259) 

2. Training 

a. Observers 

Most data cannot be collected using an automated 
collection program, and, therefore, will require the use of 
observers to be recorded. Observers must understand why 
they are collecting data, and what purpose that data will 
serve. Explaining any underlying methodologies and 
theories will empower the observer and significantly 
increase their understanding and data collection abilities. 

A training program should be created that focuses 
on when, how and where the data is to be collected. 
Observers should be familiarized with the processes 
affiliated with observation, surveys, questionnaires, and 
instrumentation, in accordance with the Data Collection and 


Analysis Plan. 

Observers 

should be given live 

scenarios 

in 

which they can 

practice. 

A proficiency 

test, 

comprised 

of 

a written exam and 

a practical 

exam. 

should 

be 


administered. The written exam should test the observer's 
knowledge of the fundamentals, while the practical exam 
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should present the observer with a scenario that will test 
their ability to correctly collect and code data. If 
necessary, a scheduled review session should be conducted. 

b. Subjects 

Military experimentation typically compares the 
current Doctrine, Organization, Training, Materiel, 
Leadership, Personnel or Facilities (DOTMLPF) with a 
proposed alternate solution. The subjects will be more 
familiar with the current DOTMLPF than the proposed 
alternative, giving it an unfair advantage. New systems 
require the greatest quantity of "familiarity training." 
Subjects need to know how they interface with the new 
system, what tasks it performs, how it performs them, how 
the system uses the underlying data, what operations the 
system performs on the data, how the data is displayed, and 
the sources of the data ( COBP for Experimentation ). 
Subjects will need to understand their roles and duties 
within the experiment. As done with the observers, a 
proficiency test should be given to test the subject's 
familiarity with the proposed DOTMLPF solution. Initial 
training of subjects and observers should proceed 
independently, but a final segment of training should be 
run with the observers gathering data on the subjects, as 
the subjects run through a scenario. 

B. CONSIDERATIONS FOR OBSERVERS 

Observers can be utilized as either silent observers 
or to administer surveys and questionnaires. Regardless, 
it is imperative that they do not influence the outcome of 
the experiment by engaging the subjects in conversation, 
expressing opinions, or interfering with the experiment in 
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any other way. They must not provide hints or suggestions, 
even when something has been overlooked. 

Incidents will occur where failure to provide 
certain information could cause the experiment to 
go drastically awry. To account for those 
instances, procedures should be established for 
the observers/controllers to get advice from the 
senior member of the observation team who, in 
turn, should confer with the experiment director 

and senior controller. ( COBP _ for 

Experimentation) 

Only experiment controllers, and not observers, should 
answer questions or offer any item of substance concerning 
the experiment. 

An observer's shift should never exceed 8 consecutive 
hours. A fatigued observer will not be able to collect 
pertinent data with the speed and accuracy of a well-rested 
observer. It is important that different observers code 
data in the same manner, so as to prevent confusion when 
the data is processed. Observer shifts should be well 
planned, and should ensure that no data is lost during 
handoff. Observer shifts should occur on different 
schedules from the subject's shift changes. This will 
allow the observer to note the process and exchanges 
between subjects. 

The Data Collection Plan should describe how 
comprehensive the observer must be and should specify which 
areas need to be covered in great detail. Otherwise, the 
observer will become overwhelmed, attempting to collect 
more data than is manageable. 

Data collectors and observers should be constantly 
supervised. Their collected data should be checked for 
completeness and confidence in its validity. Observers 
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should be discreetly alerted, without notifying the 
subjects, of events likely to affect their station. This 
will often result in the need for regularly scheduled 
meetings and timely interaction. 


C. 


DATA COLLECTION AND PROBLEM SOLVING 


Automated collection systems and instrumentation 
can fail, humans bring their own backgrounds and 
foibles to the program and, regardless of 


training, some will try to 
is interesting, rather 
assigned to collect. To 
the experiment, it is 
collectors be supervised 
mechanism is in place 
necessary data is being 
Experimentation, 268) 


collect what they feel 
than what they are 
ensure the success of 
imperative that data 
and a quality control 
to ensure that the 
collected. (COBP for 


1. Automated Data 

Automated data can be taken from screen captures, 
email archiving, requests for information, and database 
snapshots by using a simple computer program to dissect 
this data and immediately archive it. Unfortunately, 
because it is so easy to use, problems with automated data 
collection often go unnoticed. This data must be validated 
at regular intervals to ensure that the expected data is 
being processed in the expected manner. The validation 
interval should vary with respect to the data flow rate. 
Additionally, the ease with which this data can be 
collected could lead the collection team to gather large 
amounts of unimportant, unexpected data, overwhelming the 
data analysis team. 

2. Loss of Personnel 

One of the most difficult obstacles for an 

experimentation team to overcome is when a key individual 

is unable to participate in the experiment. Most commonly, 
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this lost participant was scheduled as either a test 
subject or as an observer. Whenever possible, it is a good 
idea to run extra iterations of the experiment, so that a 
fault in one run will not bring the entire experiment to a 
halt. It is also a good idea to train extra participants 
so that test subjects are replaceable. Additionally, it 
may be wise to schedule extra time to train these 
replacement subjects on the DOTMLPF solution being 
analyzed. Finally, if no other options are available, the 
experiment may have to continue shorthanded. The loss of 
manpower may affect performance, and this should be 
accounted for in the data analysis. 

In the case of an observer dropping out of the 
experiment, there are a few additional options. It would 
be convenient if there were extra observers on hand that 
could step in and fill the position. It might also be 
possible for a supervisor to take on the additional role of 
observer. It may be necessary to shift some observers from 
less important areas to fill the void. 

3. System Failures and Data Contamination 

The best solution to avoiding system failures is to 
design redundancy into the experiment and avoid single 
points of failure within the systems. All equipment should 
be tested prior to the beginning of the experimentation 
campaign. A common error with most system testing is not 
subjecting the systems to the same loads they will be 
forced to endure during the experiment, resulting in system 
failures. It is important to test the systems with the 
identical loads and traffic they will encounter during the 
experiment. 
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As with any experiment, the observation of test 
subjects will inherently cause them to work harder and 
strive to excel. Ideally this factor would be mitigated by 
the observation of teams on both the system (or idea) being 
tested, as well as the system (or idea) that is currently 
in use. Data may become contaminated due to members of the 
experimentation team influencing the outputs or even test 
subjects intentionally skewing the data. In any case, if 
data becomes contaminated, it is necessary to exclude this 
data from the analysis. 

4. Premature Reporting 

It is often very tempting to attempt to draw 
conclusions during the experimentation phase. Data 
analysis must always follow data collection and precede any 
type of reporting. The experimentation team cannot get off 
schedule and attempt to provide instant feedback on the 
results. "Instant feedback can be misleading and lead 
people to believe that they do not need to execute the data 
analysis plan" ( COBP for Experimentation , 275) . An attempt 
to provide immediate reporting tends to prevent good 
analysis and consistent results. It is very difficult to 
refocus a decision-maker's attention away from the 
immediate report, even when the data analysis proves the 
immediate report to be false. 

D. DATA ARCHIVING 

Before the data can be analyzed, it should be archived 
in its unprocessed form. Raw data will enable the 
experiment to be replicated and validated, thus permitting 
it to contribute to knowledge generation. The importance 
of preserving the raw data cannot be overstated, and it 
should be saved in more than one location and on more than 
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one medium. Additionally, all steps involved with 

processing and manipulating the data should be recorded, 
allowing the final analysis to be completely reconstructed. 

The handoff of the data between data 
collectors/observers and the data processors/analysts can 
be a major point of failure within the experiment. Both 
the data collectors and the analysts should be involved in 
the conversion of raw data into the form required by the 
analysts. 

Data [processing] should occur as soon after the 
data collection as practical, preferably in the 
days immediately following the experiment. This 
has the benefit of performing the data 
[processing] while experiment events are still 
fresh in the minds of the data collectors. It 
also has the added benefit of having all the 
collectors together so that they can work as a 
team. (COBP for Experimentation, 273) 
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X. REPORTING RESULTS/CONCLUSION 
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A. 


Figure 18 

REPORTING RESULTS 


Reporting Results 


1. Final Report 

The Experimentation team should attempt to complete 
its final report on the experiment approximately three 
months after execution. The Final Report will be a 
complete documentation of the experiment. 

a. Principle Results 

The Final Report should consist of at least 4 
major sections along with an executive summary. The first 
section should outline the principle results of the 
experiment. The principle results section should then be 
broken down into the different initiatives of the 
experiment, and each individual initiative comprising a 
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small section of the report. The layout should feature a 
short description on the conditions of experiment, followed 
by a short description of the quantity that was measured. 
This should be followed by the conclusions drawn from the 
measurement and straightforward recommendations that are 
suggested due to the outcome of this initiative. These 
initiatives should be grouped together with similar 
initiatives that fall under the same objective. 

b. Background 

Following the principle results section, the 
Final Report should now provide a brief history of the 
experiment. This should include a short history, an 
overview of activities within the experiment, a brief 
description of the experiment's overall concept, and an 
overview of the operations that took place within the 
experiment. 

c. Initiative's Context 

The initiative's context section should contain 
an overview of the conditions of the experiment quality, 
the context of the overarching focus area, and context 
descriptions of the experiment initiatives. 

d. Experimentation Status and Recommendations 

The last section of the Final Report should 

provide a general idea on the outcome of the 
experimentation campaign as a whole. It should include the 
general status of the experiment, as well as 
recommendations from each significant observation within 
the experiment. 

Also included in the Final Report should be an 
executive summary that provides a quick overview of the 
information contained within the rest of the report. 
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2. Other Reports 

a. Operational Agent Assessment 

The Operational Agent Assessment (OAA) is a 
report that is unique to the Trident Warrior program. The 
culminating step of the experimentation process is a 
Military Utility Assessment Board, where a committee of 
high-level officers makes recommendations based upon the 
results of the experiment. In order to avoid subjecting 

the MUA board to a comprehensive Final Report, the Trident 
Warrior team drafts an Operational Agent Assessment. The 
Operational Agent Assessment is tailor-made especially for 
the MUA board, and streamlines the Final Report into a few 
comprehendible points on each major focus area within the 
campaign. The purpose of this report is to inform the 

high-level decision making of the MUA board. The OAA is 
drafted by the experimentation team, and then presented, 

along with the Final Report, to the MUA board by the 
operational agent sponsoring the experiment (NETWARCOM for 
TW) . 

b. Military Utility Assessment 

Once the Final Report is complete, a Military Utility 
Assessment Board is convened to determine how the results 
and conclusions of the experiment will benefit the military 
and ultimately the warfighter. The Military Utility 

Assessment Board will publish the Military Utility 
Assessment that lists recommendations for changes to 
current Doctrine, Organization, Training, Materiel, 
Leadership, Personnel, and Facilities policy. 

Within the Navy, the completed MUAs are then briefed 
to the Sea Trial Executive Steering Group, who will rate 
each recommendation with one of four possible outcomes. 
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The first possible outcome would be to classify the 
experiment results as providing a significant enhanced 
warfighting capability impact. It would then be presented 
to the Commander of U.S. Fleet Forces Command (CFFC) for 
immediate delivery, insertion into the Naval Capability 
Development Process (NCDP), or passed on to Navy Warfare 
Development Command (NWDC). A second possible outcome for 
a recommendation from the MUA would find the experiment 
results provide a positive enhanced warfighting capability 
impact. In this case, the recommendation would be 

forwarded to CFFC, and its progress would be carefully 
tracked. A third outcome would be to classify the 

experiment results as not mature and requiring further 
examination. The event would then be rescheduled for 
additional experimentation as warranted. The final option 
would be to classify the recommendation as not providing an 
impact, and the event would then be closed out. 

B. CONCLUSION 

The overall purpose of military experimentation is to 
provide a competitive advantage to the warfighter. By 
testing new technologies and innovative methods of 
employing them, the experimentation team hopes to prove or 
disprove the added benefit of their use. An experiment is 
not deemed a success even if the MUA board does not 
recommend the solutions being tested. Success is dependent 
upon running a comprehensive and effective experiment that 
highlights the various advantages and disadvantages of 
differing military doctrine, organization, training, 
materiel, leadership, personnel, or facilities (DOTMLPF). 
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The Trident Warrior process has successfully seen 
multiple experimentation campaigns to completion. By using 
a proven, reliable experimentation process, they have 
multiplied the experimentation campaign's chances of 
success, and also increased the reliability and confidence 
of others. "I have come to rely on Trident Warrior 
information and assessments" VADM James McArthur, commander 
NETWARCOM. Enabling decision makers to make informed, 
intelligent decisions on the future of the military is 
really what it is all about. 
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APPENDIX A. TW-05 SHORT PLANNING DIRECTIONS 


Conceptually, military operational experiment planning 
is simple. An objective is defined and that leads to 
measures, then to data collection needed to provide those 
measures. The experiment results provide a determination 
of the level off success with the objective. But, the 
devil is in the details. Careful planning is needed to 
ensure that results address objectives. If you wish to 
compare the capabilities of two systems, or processes, and 
test one under one set of conditions and the other under 
different conditions, useful comparisons will not emerge. 
Successful planning proceeds along well-defined steps, each 
producing an essential planning element. This document 
explains, briefly and simply, these experiment planning 
elements. 

Planning an experiment is carried out in two phases: 

Initiative planning is concept planning. It sets the 
overarching structure for the experiment, defines the 
goals, what is to be learned and what quantities are needed 
to do it. 

Experiment planning provides experiment details: how 
it is to be carried out, what systems are to be used and 
where, specific data to be captured, where, and how that is 
to be done. 


Initiative Planning Structure 

An Initiative is described by a single sentence that 
applies to everything that is to be done under its 
umbrella. Each Initiative has a set of one or more 
Objectives, with each Objective containing the following 
components: 
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Objective 
Information Goal 

This pair of elements describes what is to be accomplished 

Questions 

They are the bridge between Information Goals and Measures and Data 

Conditions (3) 

Specific conditions may be needed to match and address the Objective. 
They and Data to be captured lead to the MSEL. 

Measures 

Data 

Quantities that must be determined to answer the Questions. 

Types of Data needed to produce the Measures 


The following will present planning structures and 
element descriptions, first with broad descriptions then 
increasing detail as the document proceeds. This 

presentation order is followed so that overarching 
understanding can be developed before a lot of detailed 
reading is done. The order of presentation is: 

• Quad Charts - Initiative Elements 

• Initiative Planning Components 

• F.I.R.E. Initiative Input 

• Threads and Numbering 

• Quad Charts to E.I.R.E. Initiative Input Mapping 

• Experiment Planning and Data Capture Plan 

• Experiment Planning Components 

• FORCEnet Attributes and Measures 

• Planning Stoplights 

• Amplifying Comments on Initiative Planning Components 

• Amplifying comments on Experiment Planning Components 


Quad Charts - Initiative Elements 

There is a single set of two Quad Charts for each 
Objective. These Charts serve the following purposes: 

• Display all pertinent information about an Objective's 
purpose and requirements. 

• Provide an easy to examine display, in a format that 
can be used for presentations. 

• Provide easily visualized information that can be 
edited and corrected before more detailed planning and 
F.I.R.E input. 

There is a Quad Chart review process where they are 
scrutinized carefully to ensure congruence across the 
planning elements and their accuracy and sufficiency. 
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A Quad Chart's elements are (content descriptions are 
later in this document): 


Header - Contains Initiative statement and POC. 

The colors here 
are chosen to 
match those in 
the following 

Objective - quadrant - One Quad set per Objective 

Info Goal - 2"^* quadrant - This is the Objective’s pair. 

Questions - quadrant - Multiple Questions allowed. 

Questions address the Info Goals. 

FIRE input 
diagram. 

Conditions - 5‘^, and 6^'’' General rather than specific descriptions. 

Yellow is used 

Measures - 7* quadrant - General descriptions of Measures types. 

Units and other specifics not necessarv. 

for three elements 
because of their 
close 

relationships. 

Data - 8* quadrant - General description of types of Data capture. 


Error! 

There may be multiple Questions, Measures, and Data 
and the associations between them is not necessarily clear 
in the Quad Charts. When information is input to F.I.R.E., 
care must be taken to provide these associations. This is 
described in the later section on Quad Charts to F.I.R.E. 
input mapping. 


F.I.R.E. Initiative Input 

Placing the Initiative information in the E.I.R.E. 
knowledge management system serves the following purposes: 

• Segment Initiative information input into experiment 
Threads that will be used for detailed planning. 

• Provide additional detail over that in the Quad Charts 
to enable Experiment Planning. 

• Archive information in a structure that provides easy 
association between elements for analysis and results 
production. 

• Provide a structure for semi-automated results 

reporting. 

The main differences between E.I.R.E information and 
the Quad Charts are: 

• E.I.R.E. segments Questions into individual Threads. 

• Conditions, Measures, and Data descriptions are more 
detailed in E.I.R.E. 

• Conditions may be more specific in E.I.R.E. to relate 
to their specific Question. 
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• Multiple Measures and Data specifications are sorted 
into those that apply to a specific Question. 


F.I.R.E. Initiative Input 


Initiative 





Objective-1 


Information Goal 




r 

Question-1 

Q-2 




Obj-2 


One sentence statement of Initiative’s Purpose during this 
experiment. There will be several Initiatives. 

One sentence statement of specific Objective Goal that 
supports the Initiative’s Purpose. 

• An Initiative will contain several Objectives. 

• Goals may relate to operational capabilities. 

Types of Information needed to address the Objective Goal. 

• Only one Information Goal statement per Objective. 

• More than one type of information may be required. 

• Includes guidelines to Conditions. 


Questions provide links between Objectives/Information and Measures. 

• Several may be needed to link to all required Measures. 

• The collective set of answers will provide the Information to be 
produced for that Objective. 

• A Question statement must lead directly to Measures. 


Cond 


V t 


Conditions: situations to must be set up to obtain the correct Information. 

• Operational - people/organizations conducting the operation 

• System - which ones and their state 

• Information - what, load, protocols, distribution, etc. 


Measures 

A, B, C... 

Meas 

D,E... 


. i 

Data 1 Data | 


:_ i 

Inputs for 

Experiment Planning 


Measures are specific quantities to be produced to answer the Question. 

• Included are appropriate units. 

• Whether they are subjective or quantitative is stated. 

Types of Data to be obtained, not details of where, when, and how. 

• Stated whether electronic, human information logger, survey. Chat 
log, etc., are to be obtained. 

• State whether subjective or quantitative. 

• Operational units at which Data is to be gathered. 


Threads and Nvunbering 

Planning Thread: a complete set of definitions. 
Initiative through Question. There can be only one 

Question per Thread , and the question is the ultimate 
definer of the Thread. 


Numbering 
Initiative Code 

Objective/Information Goal number 
Question number 


Network Initiative Example 

NET 

NET.2 2"‘* Objective in Initiative 
NET.2.3 2"‘* Objective, Question 


There is only one Initiative statement. Each of the 
Objectives within an Initiative also has a single statement 
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that is repeated for each of its Questions. A Thread's 
Question must be a single question; multiple questions are 
not allowed because proper measures can't be defined. 

Quad Charts to F.I.R.E. Initiative Input Mapping 

The diagram on the following page shows how 
information is transferred from the Quad Charts to the 
F.I.R.E. Initiative input. It is provided as a visual, 
quick reference for F.I.R.E. inputs. Descriptions of these 
planning components are in following sections. This 
section only shows the mapping from one to the other. 

As noted above, there are two Quad Charts per 
Objective. In the figure, lines with balls at the end show 
that the Objective and Information Goal statements are a 
pair. 


Same: indicates that the information in the Quads and 
FIRE are identical. This is the case for Objective and 
Information Goal. 

Expand: the information may be exactly the same, or is 
essentially the same but may need some expansion to be 
specific for that Thread. This applies to the three 
Conditions. 

Sort: the information in that quadrant has to be 

sorted to apply to its individual Thread. This applies to 
sorting Measures and Data so they are associated correctly 
with the Questions. 

• There can be multiple Questions and each of them 

defines a Thread. 

• There can be multiple Measures and they must be 

correctly associated with the Questions they will help 
answer. A Measure can apply to more than one 
Question. 

• There can be multiple Data and they must be correctly 

associated with the Measures they will produce. More 
than one type of data may be needed to produce a 

Measure. 

• The association of Question to Measures and Data can 

be one-to-many and association of Measures and Data 
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can be many-to-many. If there are too many 
associations, the Question is too broad. If 
Objectives and Questions are defined narrowly, it is 
possible that there will be only one Measure and type 
of Data for each Question. 


Quad 

Page-1 


One sentence Initiative statement 

Point of Contact 

Objective •- 

-• Information Goal 

Single overarching statement. 

Single overarching statement. 

Multiple requirements, if needed. 

Questions 

Operational Conditions 

Multiple questions, if needed, each 
supporting an Information Goal. 

-Q-1 Q-2 Q-3 

1 1 

Statement of all required conditions. 

same | 

same | 


Quad 

Page-2 


Same one sentence Initiative statement Point of Contact 

System Conditions 

Statement of all required conditions. 

Information Conditions 

Statement of all required conditions. 

Measures 

List of all Measures needed to 
answer Questions, including details. 
Ml M2 M3 M4 

Data 

List of all types of Data needed to 
determine Measures. 

D1 D2 D3 D4 D5 

_!_ 1 _!_!_!_ 


sort 


Thread Num 


Objective 


Info. Goal 


Q-1 


Oper. Cond. 


Syst. Cond. 


Info. Cond. 


Measures 

Ml 

M2 


Data 

D1 

D2 


sort 


sort 


sort 


Thread Num 


Objective 


Info. Goal 


Q-2 


Oper. Cond. 


Syst. Cond. 


Info. Cond. 


Measures 

■H M3 


Data 

D1 

D2 


I- -I 


Thread Num 


Objective 


Info. Goal 


Q-3 


Oper. Cond. 


Syst. Cond. 


Info. Cond. 


Measures 

M2 

M4 


Data 
D3 D4 
D5 


sort 


expand 


FIRE Input 
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Experiment Planning and Data Capture Plan 


Experiment Planning involves working out all of the 
details of Data capture and publishing a Data Capture Plan 
that guides conducting the experiment. The following 
figure illustrates the components of the Data Capture Plan. 


Data Capture Plan 



Architectures 

MSEL 



Systems 

Personnel 

Processes 





Data Capture Instruments 


Info Logs 


Surveys 


System 
Data Logs 


Architectures include OSDs and IDEFs. Specification 
of the systems to be used is done by a Systems Table and an 
Installs Table that tracks progress toward making the 
systems available. 

Experiment Planning Components 

Experiment Planning is grouped into categories, with 
components within each category. The following shows the 
categories and components and the type of input for FIRE 
required for entering them. Replicated means that that 
information is provided automatically from the Initiative 
inputs. Y/N refers to a Yes/No drop-down. Text refers to 
a text box where narrative input is provided. 
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Thread Information 
Thread Number 
Thread POC 
Question 
Measures 
Classified 

Fn Attrib/Meas 

(Replicated) 

(Text) 

(Replicated) 

(Replicated) 

(Y/N) 

(Text) 

This is general information about the Thread. Most of 
it is replicated. A drop-down is used to indicate 
whether the information will be classified. 

It is at this point that the connection to FORCEnet 
Attributes and Measures is made. 

Events 


Events describe those things that are to occur when 

Data Locations 

(Text) 

the data is being captured. MSEL (or other) events to 

Event Times / MSEL (Text) 

be used are listed. The exact location(s) for data 



capture are listed. 

Electronic Data 

(Y/N) 

Electronic data is that captured directly by a system 

Providing System 

(Text) 

being used in the operation or inserted to capture data. 

Data Available 

(Y/N) 

Eeasibility check. 

Collection Medium (Text) 

Needed is whether or not the data will be logged, 

Data Eormats 

(Text) 

what collection medium will be used to record it, the 

Time Stamps Avail (Y/N) 

format used, and whether synchronized time stamps 

Electronic Data POC (Text) 

will be recorded and who has direct responsibility. 

Chat Logs 

(Y/N) 

C2 and task coordination information is carried out 

Collection Medium (Text) 

via Chat. These logs often provide very useful data. 

Chat POC 

(Text) 


Observation Logs 

(Y/N) 

Personnel are used to observe operations and log 

Logger Station 

(Text) 

events. 

Forms Completed 

(Y/N) 


Loggers Assigned 

(Y/N) 

They are placed at defined locations and use forms 

Logging POC 

(Text) 

designed for their partieular data capture. 


Survey Data (Y/N) 

Surveyed Personnel (Text) 

Form Completed (Y/N) 

Administrators (Text) 

Survey Medium (Text) 

Survey POC (Text) 


Subjective opinions are gathered via surveys. 

Survey instruments that meet the Measures 
requirements are designed and administered to a focus 
group by an assigned administrator. Surveys are often 
completed using electronic capture. 


Additional Comments (Text) Any comments that are needed to clarify or direct 

any aspect of experiment conduct. 
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The normal situation in an experiment is that there 
will be data capture that applies to more than one Thread. 
Observers, instruments, logs, etc., can and should be 
shared across Initiatives and Objectives. These detailed 
planning components will reflect this in that the same 
people will appear as having responsibilities in several 
Threads. 

FORCEnet Attributes and Measures 

It is important that experiment results be mapped to 
official FORCEnet Attributes and Measures. It is also 
important the experiment measures, which will often be more 
detailed than the broader FORCEnet measures, be developed 
naturally without influence from higher-level measures. 
Thus, correlation between the two is not done until the 
Experiment Planning phase. Note in the above section that 
there is a place for entering this information. If there 
is no appropriate FORCEnet Measure, if appropriate, the 
Measuree(s) developed should be nominated for addition to 
the FORCEnet Measures. 

Planning Stoplights 

Red/yellow/green stoplights are provided for each 
Initiative. They encompass both Initiative and Experiment 
planning. There is a light for each of the major planning 
components: 


CD 

> 

O 

CD 

cc 

o 

CD 

o 

H— 

c 

o 

CO 

CD 

■D 

C 

o 

O 

0 

Q. 

■D 

C 

o 

O 

•4—< 

CO 

>^ 

■D 

C 

o 

O 

o 

H— 

CO 

0 

CO 

cc 

0 

s 

0 

_co 

0 

> 

s 

a 

O 

0 

CO 

D) 

C 

'o) 

D) 

0 

> 

o 

c 

o 

o 

C/) 

c 


Q 

LU 

LU 

o 


C/) 


The example shows all except Data of Initiative 
Planning green and Experiment planning red. 

Stoplight colors are controlled by both the Initiative 
Lead and Analysis Reviewer. Initiative colors are 

determined by whether they meet the factors: 

• Correctness of the input (meets standards for 

content) 

• Sufficiency of the input (to enable description of 
succeeding components) 

• Congruence between Initiative elements 
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Experiment planning components are judged solely on 
completeness. The Reviewer will send reasons for judgments 
to the Lead by e-mail. 


Amplifying Comments on Initiative Planning Components 

The following additional comments about the Planning 
Components are presented to better define what is required. 

Examples of Planning Components will be provided under 
separate cover for further clarification of the information 
to be provided. 

Objectives: Several types of Objective are 

appropriate: 

• Develop or enhance an operational capability. 

• Develop a system or test a system's capabilities. 

• Develop or modify a process (normally involving human 
operators). 

• Test TTP or SOP, or documents that direct operators. 

All of these, and others, are appropriate. 

The Objective statement must clearly state the exact 
goal, what is to be learned or accomplished. 

Information Goal : An Objective and its Information 

Goal are a pair. There is only one Information Goal 
statement per Objective. 

• There may be more than one type of information that is 
directly related to the Objective. 

• The statement lists the types of information needed, 
not specifics about them. They are guidelines to the 
more specific Measures and Data to be obtained. 

• The statement includes guidelines for the types of 
conditions under which the information is to be 
produced. 

Questions : A bridge is needed between the somewhat 

general Information Goal statement and the specific 
Measures to be produced. 



• A question is needed for each of the information types 
in the Information Goal. 

• Each question defines an experiment Thread. 

• Multiple Questions are not allowed, those that contain 
either explicitly or implied more than one question. 

• A Question should be stated in a way that leads 
directly to the Measures to be produced. 

Conditions : It is extremely important to have the 

proper conditions set up if appropriate results are to be 
produced. 

• Data obtained under improper conditions will yield 
results that don't answer the posed question, or 
answer incompletely, or worse yet provide incorrect 
answers. 

• Conditions lead directly to the MSEL. 

• Specified Conditions will be needed for one or all of: 
Operational, System, Information. 

Eor example: If the objective is to determine the 

maximum throughput that can be achieved with a particular 
system or process, the information load must be large 
enough so that the system can be stressed. This would be 
an Information Condition. 

Measures : Answering the Question requires that 

specific Measures be produced. These are the quantities 
that will be reported as experiment results in support of 
the answer to the question. 

• Each must be a specific quantity, with units if 
appropriate. 

• There can be more than one Measure for a Question. If 
there are several, it may be that more than one 
question is needed. Reporting will be cumbersome if 
there are too many Measures associated with a single 
Question. 
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• A Measure must lead directly to the Data acquisition 

needed to determine it. This means providing some 

specifics that provide guidance to data capture. 

For Example: "Average processing time" is 
insufficient. "Average 

time to process BDA requests by the sensor 

manager" contains the 
needed specifics. 

• If it has not been done sufficiently in Information 

Requirement, specify the operational unit, even the 
type of personnel, if they are to be involved in 
producing supporting Data. 

• The Measure definition should imply the type of 

analysis to be done to reduce the data. 

When stating Measures, care must be taken to 
ensure that needed 

Data can be captured, and analyses done, 
that will allow them to 

be determined. It is not uncommon that 

Measures are stated that 

cannot be obtained within the realities of 
an experiment. 

Data : These are somewhat general Data definitions. 

Specific details of where, when, and how they are captured 
are provided in Experiment Planning. In Initiative 

Planning, sufficient definition is provided to allow 
subsequent details to be completed. 

• Data definitions specify which types of Data are to be 
obtained: 

o system electronic data logs 
o Chat logs, etc. 
o human data loggers 

o post- or during-experiment surveys 

• These definitions alert planners to the need to 
produce particular Data collection instruments. 

Eor example: the above Measures example 
would lead to "Logs of 
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It is not necessary 


sensor requests at...", 
to specify who collects the 

logs or their format. This is done in 
Experiment Planning. 

• Correlations between Measures and Data do not need to 
be specified. They should be obvious if the 

specificity referred to above has been provided. 


Amplifying Comments on Experiment Planning Components 

Correlations between the Data, Measures to be produced 
that answer the Questions, (the Threads) are contained in 
Initiative Planning. The focus in Experiment Planning is 
on obtaining the Data. 

Experiment Planning specifies the physical things 
that must 

be done during the experiment, in sufficient 
detail, that all 

the needed systems, people, processes, and what 
they are to 

do, can be in place to produce the required Data. 

Refer in what follows to the former section on 
Experiment Planning Components. The Bold words in its 
figure are those that appear in the stoplight and provide 
the subsections below. 

Thread : Most of the information provided is 

replicated. 

• Thread numbers that are entered in the Initiative 
input will be replicated here and an input form 
provided for each Thread. 

• The POC information is not replicated because 
additional people may be involved in detailed 
Experiment Planning. 

• The FORCEnet Attribute and Measures numbers are 
entered. 

Events : The purpose of this information is to allow 

those responsible for data capture to be in place and ready 
to capture data at the appropriate times. 
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• There may be several events during the experiment that 
will provide the needed data. 

• There may be more than one data capture location, and 
different types of data may be captured at different 
locations. Specifics are needed as to what data 
where. 

• It may be necessary to identify a particular watch 
station within an organization. 

• It may be necessary to identify a particular system 
from which data is to be obtained. 

• The MSEL will be referred to for most dates and times 
for events. It the event is off-MSEL this should be 
stated along with any specifics that will help 
identify when the event will occur. 

Data : Eour types of data can be captured: 

• Electronic 

• Chat 

• Observation Logs 

• Surveys 

The section for each data type is headed by a Y/N 
dropdown. This is to indicate whether that type of data is 
required. If the answer is no, there is no need to provide 
any further input for that data type. 

The Experiment Planning input forms in E.I.R.E are set 
up so that there is no background color if no input is 
provided for an element. Thus, for a data type not 
required that whole section will be white (except for the 
Y/N answer). 

Electronic Data : These data are captured directly by 

a system being used during the experiment. 

• Each system that will provide such data must be 
identified. The specific data within the system to be 
captured must also be identified. 

• It is often the case that no automated logs are either 
available or turned on for this data capture. This 
must be determined and the Y/N is provided for status. 
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• It is important to know the medium used for data 

capture. The medium must be compatible with 

subsequent data reduction capabilities. 

• Understanding data formats is important for subsequent 
data reduction. The format should be specified and 
determination made as to whether it is compatible with 
reduction methods. 

• It is normally assumed that systems have time stamps 

available for all data output, but this often the 
case. This must be checked. Timing is crucial for 

many Measures. It must also be determined whether or 
not time synchronization procedures have been 

established. 


Chat Logs : Communications that are carried out via 

Chat can provide either direct or supporting data. 

• It must be insured Chat will be logged by the system 
being used, that personnel are assigned to capture 
Chat logs, and that appropriate recording media will 
be in place. 

• Insure that there is a means to synchronize Chat log 
times with other time logs and that logs will have 
time stamps. 

Observation Logs : Observations are logged by an 

observer assigned to a specific location. 

• The specific station to be monitored must be stated. 

• Logging forms will be specific to the information to 
be obtained at each location. Stated should be the 
specific information to be obtained and what measures 
this information will be used to produce. 

• Logs must contain the time at which a "happening" is 
observed. 

Survey Data : Surveys are the principal means for 

obtaining subjective information about a process or system. 
This information is primarily opinions. 
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• Specific personnel to be surveyed must be described. 

• Surveys will be specific to the information to be 

obtained. Stated should be the specific information 

to be obtained and what measures this information will 
be used to produce. 

• Surveys can be administered by paper or 

electronically. The means for administering and for 
post-completion archival should be stated. 

Required Documents : Most detailed planning 

information appears in other documents, MSEL, Systems, 
etc., not in the Experiment Planning forms. These other 
documents are used as references during experiment 
planning. Conversely, experiment planning also produces 
these documents. They are a crucial part of planning and 
it is not be possible to complete planning until they are 
complete. The way the planning forms are set up, it will 
not be possible to complete them until these required 
documents are also completed. 
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APPENDIX B. EXPERIMENTATION CHECKLIST 


Larry Wiener 
John Poirier 
Mark Mandeles 
Mike Bell 


January 7, 2004 

VER. 4.0 


January 7, 2004 

Experimentation Management Guide 


Purpose: This task list provides a decision support 
and management tool for experimentation sponsors, executive 
agents or action officers, and task managers. It is 
intended to support planning, preparation, and execution of 
experimentation within the context of an experimentation 
campaign. It is intended primarily as a guide to support 
individuals and teams to identify the analytical activities 
(the what's) needed to develop the experimentation 
framework, and a set of process mechanics (the how's) to 
achieve desired outcomes. It is also intended to assist 
senior decision makers in addressing tradeoffs associated 
with an experimentation campaign over extended time 
horizons, involving complex environments, multiple 
stakeholders,^ competing research interests, and limited 
resources. 


Background : This task list complements the Code of 
Best Practice for Experimentation (COBPE)^ developed by the 
Command and Control Research Program (CCRP) of the 
Assistant Secretary of Defense for C3I. The COBPE provides 
an overarching approach to the conceptualization, design, 
and execution of individual experiments or experimentation 
campaigns, and was developed to investigate evolving 
operational concepts and other areas of interest. The COBPE 
recognizes that, as a practical matter, the dynamics of 
experimentation are influenced by many factors beyond the 


^ A stakeholder is anyone who has specific interest in the results of 
the experimentation campaign. A sponsor is a stakeholder with 
specifically defined roles in providing resources to conduct the 
experimentation campaign. 

^ Code of Best Practice for Experimentation, David S. Alberts, 
Richard E. Hayes (Washington, D.C.: CCRP, 2002). 
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control of the experimenters, especially in large events 
with long planning and preparation lead times. 

The work reported here was based on the Multi-INT 
Experimental Checklist developed by a team led by Annette 
Krygiel. That effort had developed an intelligence-related 
checklist based on the COBPE. The present work represents 
an effort to extend the Multi-INT Checklist to apply to 
defense-related experimentation efforts in general. 


Format : This task list provides the elements of an 
iterative cycle (as shown in Eigure 1) combining 
development and assessment activities leading to the 
conduct of experimentation events. Critical to the success 
and relevance of each experiment or event is for the 
process to adapt to changes in priorities, environment, and 
stakeholder interests over the course of event preparation. 
There are two elements to the experiment: planning and 
execution. The planning process includes three critical 
components: 

1. Establishing an Experimentation Eramework 

2. Planning and Design 

3. Development and Validation 

Execution also includes three components: 


4. Preparation and Rehearsal 

5. Execution 

6. Analyses, Evaluation, and Transition 


It is important to note that many of the experiment 
development and execution activities are cyclical and 
overlapping rather than sequential. Eigure 1 provides one 
way to visualize this process. The process must be flexible 
enough to revisit components as needed. It should also be 
noted that as environmental conditions change and 
stakeholder interests are affected, options should be 
established for continued participation or the entry of new 
participants. The checklist is directed mainly at 
individual experiments, but the planning process described 
in Section 1 applies equally well to the planning of an 
experimentation campaign. 
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Figure 1. "The Iterative Cycle for Experimentation" 

1. Establishing an Experimentation Eramework 

1.1 Articulate and establish research/investigative 
objectives. 

1.1.1 Generate an agreed upon and commonly understood 
statement of problem(s) to be investigated. 

1.1.2 Ensure empowered representation from the 
constituents of the defined community. 

1.1.3 Ensure the objectives of the investigation or 
experiment are measurable, and are clearly, simply, 
and soundly stated. 

[These steps may be accomplished in a workshop or planning 
conference.] 

1.2 Analyze alternatives. 

1.2.1 Determine and consider alternative investigative 
methods to experimentation in terms of products, 
cost/benefit, feasibility, and risk. 

1.2.2 Determine viable approaches to investigate areas 
of interest, i.e., discrete events and linkages with 
other experiments or research activities. 

[Careful articulation of information needs can facilitate 
obtaining data from other research activities or 
confirming that a gap exists.] 

1.2.3 Determine requirements for repeatability, 
validity, and credibility. 

1.2.4 Decide to conduct the experiment (s) including 
identification of needed resources and resource 
providers. 
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1.3 Identify the community of interest. 


1.3.1 Identify stakeholders and potential sponsors as 
well as their expectations and concerns. 

1.3.2 Identify "senior leadership" for the event, i.e., 
the decisionmaker for abort criteria. 

[It is particularly important for senior leadership to be 

kept informed of the range of expectations and agendas 
represented by the various stakeholders.] 

1.3.3 Define the "total" environment - research, 
operational, political, technical. 

1.3.4 Determine range of possible effects of 
investigation at technical, operational, programmatic, 
cultural, and other levels (e.g., development of new 
knowledge, changed organizational structures, 
programmatic resources). 

1.3.5 Develop a program of peer reviews, involving 
independent evaluations by knowledgeable personnel of 
the technical, programmatic, and operational merits of 
particular aspects of the experiment or the campaign. 

1.3.6 Identify peer reviewers and enlist their 
participation. 

1.4 Identify interests of stakeholders. 

1.4.1 Identify points of common interest and points 
where interests do not converge, in order to define a 
deconfliction process. 

[The experimentation campaign planners must accommodate the 
concerns of those who will be affected by the 
experimentation process and its results.] 

1.4.2 Define the level(s) of interest for process 
results—national, local, service, joint, limited 
objective, campaign. 

1.4.3 Define, with senior leadership, the purpose and 
desired product types for the process, to avoid 
excessive focus on the output of only one event or 
experiment. 

1.4.4 Establish strategic, campaign level focus linking 
multiple events. Emphasize cumulative development of a 
body of knowledge that does not depend on the results 
of only one experiment or research activity. 

1.5 Engineer flexibility into the process. 

1.5.1 Define critical decisions (rather than 

milestones) to allow for confirmation of sponsor and 
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stakeholder interests, commitment to the course of 
action, and opportunities for the graceful exit of 
stakeholders who no longer wish to be part of the 
event as well as entry of new stakeholders. 

1.5.2 Determine constraints on the experiment or on the 

campaign that impact iteration/refinement cycles (see 
Figure 1). 

[Reviews should be provided to senior leadership to ensure 
that the ramifications of decisions and directions are 
recognized.] 

1.6 Scope the effort. 

1.6.1 Obtain general agreement on the type of 
experiment(s) to be conducted e.g., discovery, 
hypothesis testing, and demonstration,. 

[In principle, it may be advisable to change the experiment 
type during the development process. This is a major 
decision requiring careful assessment and stakeholder 
participation.] 

1.6.2 Derive the simplest experimentation approach 

lab test, modeling and simulation, limited field test, 
etc. - to deliver desired results. 

1.6.3 Document the responsibilities of each participant 
and obtain a common understanding and agreement among 
the participants. 

1.6.4 Identify and execute any required contractual 
vehicles. 

1.6.5 Determine, plan, and program to ensure 
availability of required prototypes, systems, 
materiel, databases, and infrastructure. 

1.6.6 State desired outcomes, being explicit in terms 
of goals, metrics, applications, and relevance to type 
of experiment. 

1.7 Establish strategic framework. 

1.7.1 Identify, assess the relevance of, and pursue 
cooperative joint efforts with complementary research 
programs including those in military services/joint 
environments, government agencies, academe, research 
and development organizations, and other venues. 

1.7.2 Develop contingency planning options and 
procedures in anticipation of unexpected events and 
disruptions. 

1.7.3 In the case of incremental funding, ensure 
resource commitment and scheduling. 
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1.7.4 Establish tracking and management mechanisms, 
e.g., funding expenditures and calibrate against 
projected costs for the experiment, development of 
needed infrastructure and capabilities, task 
execution. 

1.7.5 Develop an experiment schedule to address all 
phases and resource requirements. 

1.7.6 Assess and account for potential impact of 
experimentation environment on experimentation process 
and outcomes. 

1.7.7 Initiate actions to secure required funding, 
resources, and approvals. 

1.8 Identify requirements. 

1.8.1 Determine minimum resources (personnel, 
facilities, time) to execute the experiment. 

1.8.2 Identify unique requirements for experiment 
conduct and design (infrastructure) to include: 

Data collection team qualifications/assignments. 

Data collection team training. 

Instrument development. 

Consistency of involvement in experiment preparation 
period. 

Selection of experimentation infrastructure 
(live/modeling and simulation supported). 
Identification of all factions to be represented, 
e.g., Blue/Friendly, Red/Hostile, White/Neutral, 
Green/Third Party, etc. 

1.8.3 Identify unique requirements for experiment 
participant (subject) selection and preparation to 
include desired expertise, training requirements, 
preparation timelines, language proficiency, 
experience sets, and Service or Combatant Command 
representation. 

1.8.4 Determine required approvals to conduct 
experiments, e.g., review board approvals for use of 
human participants in experimentation. 

1.8.5 Schedule a policy review early to identify 
security issues, such as need for accreditation, or 
policy waivers. 

1.8.6 Determine requirements for, and initiate 
processes to obtain, waivers, such as that of security 
policy. 

1.9 Commit resources. 
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1.9.1 Gain commitment of research sponsor (s) and degree 
of support through written agreement with 
participating agencies, i.e., ensure proper staff 
availability and secure needed resources and/or 
funding. 

1.9.2 Assign and ensure commitment of Experiment 
Director/Action Officer through the research sponsor. 

1.9.3 Document requirements in experiment objectives, 
requirements, and expected outcomes with research 
sponsor. 

1.9.4 Ensure early identification and commitment of all 
experiment participants by their respective 
organizations - customers, process-owners, subject 
matter experts, prototype developers, facilities- 
owners, trainers, security personnel, etc. 

1.9.5 Document facility requirements and obtain 
agreements for their provision with hosts and 
supporting organizations to ensure scheduling and 
availability. 

2. Planning and Design 

2.1 Define role of the particular experiment within the 
campaign 

2.1.1 Assess current state of knowledge in terms of 
what the campaign has answered so far and identify 
remaining unresolved issues 

2.1.2 Inform senior leadership and confirm research 
priorities. 

2.1.3 Define and decide the next step in 
experimentation or investigation. 

2.2 formulate experiment. 

2.2.1 Review results of research relevant to the 
intended experiment, including reviews of studies, 
experiments, customer and process-owner input, etc. 

2.2.2 Identify the products and implications of the 
experiment, such as residual assets, evaluation, 
requirements for acquisition, new business processes, 
etc. 

2.2.3 Ensure that the experiment incorporates 
appropriately mature elements, such as existing 
prototype versions, surrogates or mock-ups, and 
modeling and simulation. 

2.2.4 Submit the experiment plan for peer reviews or 
independent assessments. 
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2.2.5 


Develop hypotheses (IF, THEN statements) that are 
generated and structured to contribute to knowledge 
about the concept, capability, system, or process 
being investigated. 

2.2.6 Assess prototypes, cost, risk, and schedule 

acceptability constraints (for experimentation). 

2.3 Plan the analysis and evaluation methodology. 

2.3.1 Identify control variables and determine means to 
treat them adequately. 

[All variables should be identified by type (independent, 
dependent and control). Both manipulation and control 
are integral to hypothesis-testing experiments. The 
dependent variables are observed systematically under 
specified conditions while the factors considered to 
cause change - the independent variables - are varied. 
Other potentially relevant factors must be held 
constant, either empirically or through statistical 
manipulation. (Extracted from Code of Best Practices 
for Experimentation by Dr. David S. Alberts.)] 

2.3.2 Determine whether a documented baseline for 
comparison exists or the means to generate one in time 
for the conduct of the experiment. 

2.3.3 Ensure that the evaluation method selected is 
relevant to the type of experiment, select appropriate 
metrics, and determine options to populate them 
through data collection. 

2.3.4 Ensure participants for experiment have desired 
expertise, training requirements, preparation 
timelines, language proficiency, experience sets, and 
Service or Combatant Command representation. 

2.3.5 Ensure processes and organizational changes are 
considered in the evaluation. 

2.3.6 Ensure that the plans for scenario generation are 
sufficiently timely and that they span the conditions 
of interest to the sponsors. 

2.3.7 Develop Analysis Plan, framework, and reporting 
requirements. 

2.3.8 Develop data collection plan(s) for processes 
during the experiment and/or across the range of 
experiments within the campaign to ensure sufficient 
data to support the evaluation. Consider means to 
ensure that: 

Data collected reflect critical indicators. 

Quality control processes are included 
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Data collection allows for re-use and archiving 
through file collection and indexing 
Standard data formats are used where possible, 
including date/time references. 

The evaluation plan addresses how to interpret, 
generalize, or scale the results of the experiment. 

2.3.9 If modeling and simulation will be used to 

validate and expand experiment findings, then ensure 
that appropriate models and simulation capabilities 
are available, or planned. 

2.4 Plan experiment and develop experimental architecture. 

2.4.1 Determine and ensure understanding of applicable 
technical standards that must be developed and/or 
applied where feasible. 

2.4.2 Determine if existing experimentation 
infrastructure in other experimentation and research 
venues such as Joint Forces Command (JFCOM), service 
laboratories, and academic institutions can be 
used/leveraged. 

2.4.3 Identify and describe any legacy system 
enhancements or needed infrastructure/tool 
development. 

2.4.4 Identify, schedule, and ensure commitment by 
their owners and/or sponsors of required prototypes, 
systems, materiel, databases, and infrastructure, 
including all Government Furnished Equipment (GFE) and 
Government Eurnished Information (GEI) by specific 
dates. 

2.4.5 Ensure appropriate means of technical control 
over infrastructure development, such as configuration 
management processes. 

2.4.6 Ensure engineering resources are available as 
needed to develop the end-to-end architecture to 
support the experiment. 

2.4.7 Map appropriate elements of operational concept 
or problem being investigated to scenario. State 
research questions and framework for reporting results 
per the analysis plan. 

2.5 Conduct facility planning. 

2.5.1 Determine and schedule facilities and resources 

required for experiment in terms of manpower, 
equipment, infrastructure, etc. in accordance with the 
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needs of each specific phase of the experiment and by 
specific dates. 

2.5.2 Identify and resolve potential conflicts with 
other events supported by the experimentation facility 
such as training, integration and test, rehearsal, and 
experiment conduct. 

2.5.3 Assess facility layout and determine means to 
minimize or avoid distractions and disruptions, such 
as from visitors and observers. 

2.6 Develop training. 

2.6.1 Determine training criteria and required 
standards of proficiency. 

2.6.2 Plan and program training for experiment support 
personnel including observers, roleplayers, data 
collectors, M&S support, collection managers and 
others. 

2.6.3 Plan and program training for experiment 
participants to ensure familiarity (as appropriate) 
with operational concepts, experiment infrastructure, 
experiment information exchange processes, and roles 
of other participants. 

2.7 Conduct security planning. 

2.7.1 Identify security issues and/or engineering 
requirements using appropriate instructions, e.g.. 
Defense Information Technology Security Certification 
and Accrediation Process (DITSCAP) or National 
Information Assurance Certification and Accreditation 
Process (NIACAP). 

2.7.2 Assess impact of security requirements or 
experiment and data availability, e.g., access by 
foreign nationals, "uncleared" researchers, storage of 
data and documents, system connectivity. 

2.7.3 If needed, secure and schedule needed security 
resources, e.g., storage containers, destruction 
resources, access control resources, storage devices, 
etc. 

2.8 Determine risk and define risk management procedures. 

2.8.1 Identify types and levels of risk to experiment 

success, infrastructure, personnel availability, 
funding, schedule, and cost 


104 



2.8.2 Determine requirements and options to mitigate 
risk. 

2.8.3 Develop risk mitigation plans for high-risk 
elements such as immature infrastructure components, 
use of surrogates, variance in software/hardware 
versions, long lead items, and generation of essential 
databases. 

2.8.4 Establish guidance on documentation requirements 
of any design and development. 

2.9 Conduct schedule planning and control. 

2.9.1 Ensure that the proposed schedule is viable given 
the scope of the experiment, the identified risks, and 
the proposed funding. 

2.9.2 Identify critical event dependencies and long- 
lead items for key activities. 

2.9.3 Schedule: 

Adequate time for development and conduct of training 
for observers, support staff, as well as participants. 

- Progress reviews for assessments of risk and progress 
for experiment participants, through all phases of the 
experiment. 

- Progress reviews, peer reviews, and critical decision 
reviews for programmatic goals, including one at the 
completion of planning phase. 

Sufficient time for testing and integration of the 
hardware/software/infrastructure. 

2.10 Conduct transition planning. 

2.10.1 Develop a transition plan to pass findings and 
conclusions to stakeholders. 

2.10.2 Implement a communications strategy to 
disseminate information to participating and 
interested parties, e.g., form/type of briefings, web 
posting, email distribution lists, etc. 

2.10.3 Identify budget or Program Objective Memorandum 
(POM) submissions potentially affected by the 
experiment. 

2.10.4 Identify and schedule a knowledge repository— 
e.g., an archive or website—for experimental results. 

2.10.5 Ensure agreement and funding support for any 
residual assets either left in place or transitioned 
to the appropriate sponsors. 

2.10.6 Estimate funding requirements for follow-on 
research and development activities, e.g., refinement 
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of prototype, additional experiments, acquisitions, 
etc. 

3. Development and Validation 

3.1 Complete design and implementation plan. 

3.1.1 Schedule or complete final experiment design. 

3.1.2 Ensure final experiment architecture and 
implementation are on track. 

3.1.3 Ensure development of the data collection plan is 
on schedule or complete, including quality control 
processes. 

3.1.4 Ensure test and integration is proceeding on 
schedule, or completed. 

3.1.5 Schedule and/or complete development and testing 
of infrastructure, support tools, required databases. 

3.1.6 Ensure experiment analysis and evaluation plan 
are complete and all activities are on track. 

3.1.7 Establish and determine application of measures 
of effectiveness, metrics, and/or success criteria and 
incorporate in the analysis and evaluation plan. 

3.1.8 Eor experiment/research campaigns, establish 
iterations and entry/exit criteria. 

3.1.9 If required, prepare help desk for the 
experiment. 

3.1.10 Ensure security policies reviews are completed, 
accreditations obtained, or formal waivers obtained. 

3.1.11 Ensure that development and implementation of 
operational scenarios, concepts, and script are 
complete or proceeding on schedule, including 
Models and simulations. 

Training. 

Orientation and familiarization plans and materials 
for participants. 

3.2 Execute readiness review. 

3.2.1 Schedule review of experiment preparations for 
the research sponsor(s). 

3.2.2 Identify a means to address problems and issues, 
and ensure availability of resources for corrective 
actions. 

Ensure data collection instruments and participant 
forms/guestionnaires are available. 

Schedule observer interviews. 
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Schedule appropriate mechanisms to capture and 
disseminate information such as recording of 
briefings. 

3.2.3 Implement help desk. 

3.2.4 Establish visitor access and control procedures. 

3.2.5 Ensure data collection team and resources are 
ready, including a means to archive all collected 
data. 

3.2.6 Implement process to capture Lessons Learned. 

3.2.7 Implement control processes for changes to 
experiment environment, infrastructure, and procedures 
that may be caused by anomalous interruptions. 

3.2.8 Describe and document rehearsal procedures for 
all participants and assign all responsibilities. 

3.2.9 Ensure sufficient time is included in the 
schedule to take corrective action as needed after the 
rehearsal. 

3.2.10 Ensure all activities in the conduct of the 
experiment are addressed and documented, including 
those of participants, observers, and visitors, in 
addition to equipment, systems, and infrastructure. 

4. Preparation and Rehearsal 

4.1 Conduct rehearsal. 

4.1.1 Include participants, observers, support 
personnel, and (acting) visitors engaged in the 
rehearsal. 

4.1.2 Exercise the supporting infrastructure, including 
instrumentation. 

4.1.3 Exercise all key activities and processes in the 
experiment design and plan- including data collection 
and analysis. 

4.1.4 Ensure rehearsal includes practice vignettes and 
end-to-end scenarios. 

4.1.5 Stress the system architecture to ensure all 
experiment requirements are supported. 

4.1.6 Ensure a process exists to capture anomalies and 
unexpected disruptions and to take corrective actions 
in the experiment. 

4.1.7 Verify that the experiment addresses the original 
objectives. 

5. Execution 

5.1 Collect data. 
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5.1.1 


Collect, verify, and archive data including 
observer notes and informal interviews. 

5.1.2 Monitor quality control mechanisms for data 
collection. 

5.2 Ensure daily communications plan is followed. 

5.2.1 Schedule roundtable information exchange meetings 
of the data collection team and other observer groups. 

5.2.2 Archive notes, data collection instruments, and 
other materials such as emails, voice communications 
logs generated during experimentation events. 

5.3 Document experiment process and lessons learned. 

5.3.1 Document changes (functionality) or deviations 
from experimentation plan in accordance with technical 
control plan. 

5.3.2 Record down time and perceived impact on 
experiment in light of system or process failures such 
as power outages, interruptions, and loss of 
participants in logs and data compilation. 

5.3.3 Capture and document lessons learned in 
experiment process. 

5.3.4 Provide "hot wash" briefing and discussion for 
participants and capture resulting comments and 
observations. 

6. Analyses, Evaluation, and Transition 

6.1 Provide a "quick look" report and briefing for 
stakeholders. 

6.1.1 Report, at a minimum, the assumptions and major 
findings of the experiment and any initial 
recommendations affecting the experiment campaign or 
related operations. 

6.1.2 Identify any unresolved issues, uncertainties, 
and sensitivities discovered in the experiment. 

6.1.3 Provide for the widest possible circulation of 
the briefing and report and invite review and 
critique. 

6.1.4 Caveat "quick reaction" results to avoid 
premature conclusions and recommendations. 
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6.2 Prepare and distribute a report of preliminary 

findings within a reasonably short time after the 
experiment. 

6.2.1 Distinguish clearly between findings (factual 
observations and data) and interpretations of the 
results. 

6.2.2 Describe the effects of interruptions, 
disruptions, anomalies, etc. 

6.2.3 Collect peer review results and incorporate them 
into revised and future reports. 

6.3 Prepare and publish formal reports. 

6.3.1 Adopt and implement a publication and 
dissemination plan to provide a range of products 
(decision papers, summary reports, briefings, 
scientific papers, articles, and books) suited to the 
needs of various audiences and stakeholders. 

6.3.2 If appropriate, provide a synthesis of findings 
and interpretations from across several related 
experiments, especially those in the experimentation 
campaign that includes the present experiment. 

6.3.3 Provide recommendations for iterations of the 
experiment or practical application of results in 
exercises, as well as future experiments based on 
issues uncovered or knowledge derived. 

6.3.4 Provide lessons learned about the experimentation 
processes, tools used, and infrastructure. Incorporate 
considerations for scaling or expanding results by 
further experimentation, modeling and simulation, or 
other means. 

6.3.5 Identify and address programming and budgeting 
implications, including resources needed to transition 
results, e.g., refinement of prototypes, more 
experiments, implications for on-going acquisitions. 
Identify any budget or POM submissions affected. 


6.4 Archive experiment design, data, and results for 
future use. 

6.4.1 Collect all data records, interview transcripts, 

scenario descriptions, training manuals, and other 
artifacts immediately after execution of the 
experiment and preserve them in their original form. 


109 



6.4.2 


Compile and archive a dietionary/glossary of the 
terms, constructs, and acronyms used in the 
experiment. 

6.4.3 Compile and archive a dictionary of methodology 

and metrics (including definitions and scales) used in 
the experiment. 
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APPENDIX C 


COMFLTFORCOM INSTRUCTION 3900.lA 


CFFCINST 3900.lA 
N8 

22 Dec 03 

COMFLTFORCOM INSTRUCTION 3900-lA 

Subj; SEA TRIAL 

Ref: (a) Naval Power 21 

(b) CNO 011515Z Oct 01 {Sea Power 21 Implementation) 

Enel: (1) Sea Trial process diagram 

1. Purpose ■ To provide Commander, U.S. Fleet Forces Command 
(CFFC) policy guidance, establish responsibilities, and 
prescribe general procedures for establishing and executing Sea 
Trial in support of Sea Power 21. 

2. Cancellation: COMFLTFORCOMINST 3900.1 

3. Background. Naval Power 21, reference (a), provides 
SECNAV's vision for Navy and Marine forces. CNO and CMC have 
established Sea Power 21 and Marine Corps 21 as the Navy and 
Marine Corps' strategy for the 21°'^ Century. Sea Power 21 is 
built around the core concepts of Sea Strike, Sea Shield, Sea 
Basing, enabled by FORCEnet, with US Marine Corps Expeditionary 
Maneuver Warfare (EMW) and Joint maneuver warfare from the Sea 
Base fully embedded. Sea Trial will integrate emergent concepts 
and technologies into an experimentation process that will 
produce continuous improvements in naval warfighting. Reference 
(b) established CFFC as the lead agent for Sea Trial. 

4. Objectives. Through Sea Trial, Fleet Forces Command (FFC) 
will formalize and fully integrate concept development and 
technology insertion into the experimentation process. Sea 
Trial will be Naval in focus through close coordination with the 
Marine Corps Combat/Force Development Process, and will support 
concurrent development and testing of joint warfare capabilities 
from the Sea Base. A comprehensive Sea Trial Concept 
Development and Experimentation (CD&E) Plan will rapidly mature 
Sea Shield, Sea Strike, Sea Basing and FORCEnet concepts and 
technologies, codify experiment results in doctrine, and reflect 
in programs of record. The CD&E Plan will be concept-based and 
aligned to mission capabilities gaps and Fleet priorities. 
Emphasis will be on: 
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a. Speed -- Expeditious induction of new concepts and 
technologies into the experimentation process, along with rapid 
transition of promising outcomes into doctrine and program. 

b. Comprehensive -- Single point visibility over all Navy 
experimentation other than the Discovery and Invention portion 
of the Navy's Science and Technology effort. 

c. Jointness -- Maximum incorporation of Joint Concept 
Development and Experimentation {JCD&E) into Navy processes, 

d. Relevance -- Applicability of experimentation to mission 
capability gaps in naval warfighting. 

e. Economy -- Maximum effectiveness and efficiency from 
Concept Development and Experimentation (CD&E) efforts. 

5. Joint Applicability. Marine Corps operating forces are part 
of the fleet. Accordingly, the Marine Corps will be an equal 
partner at each level within the Sea Trial process and will use 
the Sea Trial process to coordinate their (CD&E) efforts that 
are inherently naval and not service specific. As the Deputy 
Commandant for Combat Development, Commanding General Marine 
Corps Combat Development Command (CG MCCDC) is the Marine Corps' 
Sea Trial coordinator. Sea based operations will be 
Increasingly important to overall j oint/coicibined warfare 
missions. To that end, coordinated efforts with the Army, the 
Air Force and other forces are important to maximize the payoff 
of ongoing naval force experimentation to future joint/combined 
war fighting effectiveness. Concurrently, and to the greatest 
extent possible. Sea Trial will also support future coalition 
operations. Finally, Joint Forces Command (JFCOM) plays a key 
role in JCD&E, and a strong relationship will exist between Sea 
Trial efforts and JFCOM interests. 

6. Sea Trial Organization. Sea Trial is a CFFC-led 
collaborative effort that includes COMPACFLT, COMNAVEUR, DCNO 
{N6/N7), CNR, CNWDC, CG MCCDC (DC, CD), C3F, C2F, C5F, C6F, C7F, 
NETWARCOM and others involved in Naval CD&E 

a. Sea Trial Executive Steering Group (STESG). CFFC will 
establish and chair a flag officer Sea Trial Executive Steering 
Group (STESG) comprised of COMPACFLT, COMUSNAVEUR, DCNO (N6/N7), 
CNR, CNPiDC and CG MCCDC (DC, CD) . COMSECONDFLT, partnered with 
COMFIFTHFLT and COMSIXTHFLT, and COMTHIRDFLT, partnered with 
COMSEVENTHFLT, will participate as Sea Trial Operational Agents 
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for Sea Strike, Sea Basing and Sea Shield. COMNAVNETWARCOM will 
participate as the Sea Trial Operational Agent for FORCEnet. 
Marine Corps operational forces (marfORLANT and MARFORPAC) 
designated by CG MCCDC (DC,CD) will participate as counterparts 
to the Operational Agents to address Naval issues as 
appropriate. A single Flag-level representative of each SYSCOM 
will participate in STESG meetings as advisory members for 
matters relating to Research and Development (R&D) and 
industrial base capability. A COMOPTEVFOR flag level 
representative will participate in STESG meetings and advise on 
Operational Test and Evaluation (OT&E) matters. The STESG will 
coordinate an agile, responsive and transfonaational Sea Trial 
process. Members and designated commands will coordinate 
quarterly to validate recently completed military utility 
assessments, determine if additional testing is required and/or 
recommend follow-on action including analysis and programming 
via the Naval Capabilities Development Process and PPBE, approve 
the Sea Trial Execution Plan including any modifications, 
additions or deletions of follow-on events based on the results 
of other cortpleted testing, resolve issues of resources and/or 
priorities, assess non-traditional solutions to ongoing 
warfighting development Issues, and make recommendations to CFFC 
on the viability of emergent concepts and technologies. E/vents 
approved by the Operational Agent but remaining unscheduled will 
be reviewed at each STESG, This dynamic review process will 
ensure Sea Trial remains agile and responsive to emerging 
concepts or materiel solutions to warfighting challenges- 

b. Operational Agents (OA)■ COMSECONDFLT (partnered with 
COMFIFTHFLT and COMSIXTHFLT) Is the Sea Trial Operational Agent 
for Sea Strike and Sea Basing. COMTHIRDFLT (partnered with 
COMSEVENTHFLT) Is the Sea Trial Operational Agent for Sea 
Shield. COMNAVNETWARCOM is the Sea Trial Operational Agent for 
FORCEnet. Operational Agents are responsible for overall 
prioritization and coordination of all aspects of warfighting 
CD&E within their respective pillar areas. The Operational 
Agents, supported by members of the Fleet Collaborative Teams 
(FCT) and NWDC, will validate proposed CD&E initiatives 
(contained in the Sea Trial Information Management System 
(STIMS)); oversee the planning, coordination and conduct of Sea 
Trial events; and brief results to the STESG. Any activities 
desiring to conduct a Sea Trial event that falls in part or 
wholly under one or more of these pillars will fully coordinate 
such initiatives through the respective OA, and will ensure 
these initiatives are entered into the STIMS database. Marine 
Corps operational forces (MARFORLANT AND MARFORPAC) designated 
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by CG MCCDC (DC,CD) will participate as counterparts to the 
Operational Agents to address Naval issues as appropriate. 

c, Fleet Collaborative Teams (FCTs) ■ These teams are 
chartered by CFFC and will contribute to the Naval Capability 
Development Process (NCDP) and CD&E efforts. FCTs will leverage 
the intellectual capital of Warfare Centers of Excellence 
(WCOE), NWDC, ONR, TYCOMs, SYSCOMs, Fleets and Other commands, 
and are generally organized along Mission Capability Package 
(MCP) lines under the Naval Capability Pillars (NCP) of Sea 
Strike, Sea Shield, Sea Basing and FORCEnet. FCTs will provide 
the Operational Agents with the necessary expertise to develop 
and evolve the Sea Power 21 operating concepts, WCOEs are key 
to the CD&E aspect of the FCT's charter, as they will be 
responsible for key elements of doctrine, tactics, techniques 
and procedures (TTP), Lessons Learned Remedial Action, and CD&E. 
This directive authorizes Operational Agents, acting on behalf 
of CFFC, to direct FCTs in their MCP areas for development and 
implementation of experimentation plans. In addition to their 
role in the NCDP process, FCTs will contribute to CD&E in the 
following manner: 

(1) Develop Sea Power 21 Operating Concepts. The FCTs 
are responsible to their respective Operational Agents for 
development and evolution of Sea Power 21 operating concepts and 
the overall direction for CD&E in tlieir respective areas. 

NWDC will coordinate this effort across all OA Sea Trial 
activities, and FFC N8 will assist in this process. 

(2) Provide Subject Matter Experts (SMEs) for, and 
participate in, concept development initiatives germane to their 
mission area to include wargaming (including modeling and 
simulation), MCP Integrated Product Teams (IPT), Sea Trial 
Military Utility Assessment (MUA) teams, limited objective 
experiments (LOEs), employment of operational prototypes and 
major field experiments (e.g. Fleet Battle Experiments (FBEs), 
and USKC Advanced Warfighting Experiments (AWEs)). Operational 
Agents will periodically convene concept development workshops 
that combine the efforts of SMEs, operators, technologists, and 
network specialists to formulate innovative new operating 
concepts that harness advanced systems to provide enhanced 
capabilities in their assigned mission areas. 

(3) In cooperation with and with the approval of the 
OA(s), develop Sea Trial Execution Plans that support the Sea 
Trial CD&E Plan, FCTs will develop Execution plans for their 


4 


114 


CFFCINST 3900.1A 
N8 

mission/functional area for approval by the Operational Agent in 
coordination with CFFC (NWDC facilitating). 

(4) Ensure Sea Trial Events are properly planned and 
scheduled. The FCT, assisted by NWDC, will identify 
requirements necessary for Fleet schedulers/planners. 

(5) Review and Process Military Utility Assessments. 

(6) Participate in TACMEMO development and support 
experimentation and Tactics, Techniques and Procedures (TTP) for 
validated capabilities. 

(7) Oversee and report to their respective Operational 
Agents on the implementation of the approved and funded CD&E 
activities laid out in the Sea Trial Execution Plan. 

7. Sea Trial Process. 


a. CD&E initiatives, whether concept or technology based, 
will be submitted by the FCTs, OPNAV, ONR, SYSCOMS, Industry or 
other agencies to NWDC for consolidation, tracking, and 
coordination via the STIMS database. Demonstrations, events, 
experiments, or prototypes designed to explore new technologies 
or concepts with the fleet, or on behalf of the fleet, require 
strict adherence to the Sea Trial process. This includes 
entering information into STIMS. MWDC will ensure 
distribution to all activities. 

b. The OA, assisted by NWDC and FCTs, will examine CD&E 
initiatives to determine their technical feasibility, relevance 
to the Sea Trial CD&E Plan, whether the initiative meets Fleet 
priorities (as determined by OAs) and/or Mission Capability gaps, 
and Identify the appropriate experimental venue. Initiatives 
failing to meet the criteria will be rejected by the Operational 
Agent and the sponsor and CFFC will be informed of the rationale. 

c. The Operational Agent will exeimine validated initiatives 
to determine if additional monetary resources are required. 

(1) Validated initiatives that are adequately resourced 
will continue to be processed as follows. The event's sponsor, 
in coordination with the FCTs and NVJDC, will develop an 
experimentation plan and submit it to the OA for approval. This 
plan will describe how the experiment will be conducted and will 
include a data collection and analysis plan (DCAP) with 
appropriate metrics. Once approved, these validated initiatives 
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will become Sea Trial events and the OA will provide an 
information brief to the STESG at the first opportunity. CFFC 
{with NWDC as agent) will then assign the Military Utility 
Assessment (MOA) team and MUA timeline. 

(2) Initiatives with resource issues will be forwarded 
to NWDC for broad effort resolution, or (if beyond existing 
resource capability) for presentation by NWDC to the next STESG 
for resolution (i.e. funding, postponement, modification, or 
termination). In those instances where an important Sea Trial 
opportunity will be missed by waiting for the next full STESG 
meeting, NWDC will prepare and immediately electronically 
forward to all STESG principals a summary of the proposed event 
and actions required to support a coordinated decision, ensuring 
the responsiveness of the overall system. 

d. NWDC will align all events across all Naval Capability 
Pillars and present a consolidated Execution Plan to the STESG. 
The Execution Plan will be presented to the STESG in June and 
will outline the CD&E intentions for the subsequent two fiscal 
years. Changes to the Execution Plan will be briefed to the 
STESG by the OA proposing the change. 

e. The OA will publish a Quick Look upon completion of the 
Sea Trial event and provide it to the STESG. 

f. The MUA team will complete the MUA in accordance with the 
MUA timeline and submit it to the OA for approval. Completed 
MDAs will be distributed and briefed to the STESG by the 
respective OA. NWDC will coordinate these briefings for 

the STESG, with the goal of providing an efficient summary of 
Sea Trial products completed since the previous STESG meeting, 
and as a basis for review of planned and pending Sea Trial 
events in support of dynamic validation of the overall Sea Trial 
CD&E Plan. MUA recommendations will be categorized into 
Doctrine, Organization, Training, Materiel, Leadership, 

Personnel and Facilities format. The STESG will rate each 
recommendation based on four possible outcomes. OPNAV N6/N7 is 
the single point of entry for FFC products (with the exception 
of doctrine) resulting from Sea Trial experiments. Events 
classified as having significant enhanced warfighting capability 
impact will be presented to CFFC for immediate delivery to OPNAV 
N6/N7 for insertion into the NCDP or forwarded by N6/N7 to the 
applicable OPNAV process agent for action (e.g. Nl (Manpower and 
personnel), N4 (Readiness, material, facilities), NOOT 
(Training)). Those items that are doctrinal related will be 
passed by CFFC directly to NWDC. Events classified as having 
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positive enhanced warfighting capability impact will be 
presented to CFFC for delivery to OPNAV N6/7 and distributed as 
mentioned above. CFFC will carefully track the progress of 
these inputs. Events evaluated as not mature enough and 
requiring further examination will be rescheduled for additional 
experimentation as warranted. Events evaluated as not providing 
an impact will be closed out. 

g. The status of all Sea Trial events will be maintained on 
STIMS and will be available for stakeholder review. This status 
will be provided to the STESG as a read ahead before meetings 

8. Sea Trial Concept Development and Experimentation (CD&E> 

Plan. The Sea Trial CD&E Plan will be directed by CFFC and 
drafted by NWDC based upon the inputs of the Operational Agents. 
The CD&E Plan will be focused 10 years out and contain a detailed 
listing of Fleet priorities and mission capability gaps. This 
CD&E Plan will be a dynamic document, that will establish metrics 
to assist OAs in determining applicability of their Sea Trial 
efforts and allow the STESG to measure overall Sea Trial 
progress, support proper planning for complex testing 
requirements, and take full and increased advantage of emergent, 
smaller scale, more focused testing opportunities. The CD&E Plan 
should implement a comprehensive Sea Trial Roadmap that 
integrates studies, war games, experimentation, and exercises 
with proposed evaluation metrics. This CDtE Plan should include 
those promising concepts and technologies that speed Sea Power 21 
capabilities to Che Fleet. 

9. Sea Trial Execution Plan. The Sea Trial Execution Plan is a 
two-year document and represents a collection (synchronized 
across NCPs) of OA planned Sea Trial events. The document will 
be prepared by MWDC based on inputs from the OAs and briefed to 
the STESG for approval each June. The Execution Plan will also 
contain evaluation metrics and an execution timeline. 

Subsequent changes in OA planning requiring a change to the 
Execution Plan will be briefed by the individual OA. 

10. Sea Trial Information Management System (STIMS). STIMS, an 
NVJDC developed and maintained interactive web based database, 
will be the central library of initiatives and technologies and 
will be used as the tool to manage Sea Trial events and related 
activity. Operational Agents, supported by all FCTs and all 
other Sea Trial event sponsors are responsible for populating 
and updating all STIMS database entries for their respective 
efforts. Other commands, including CNR, OPNAV, SYSCOMS, etc.. 
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will ensure all experimentation is fully documented in STIMS. 
The initiative submission format will be contained in STIMS 
and entry into STIMS will be the minimum essential requirement 
for any event to be considered for future Sea Trial status * 

KWDC will maintain business rules that will be used by CFFC for 
utilization of STIMS by all stakeholders. 

11. External Sea Trial Interfaces. Sea Trial must be 
synchronized with other processes related to CD&E. CFFC {with 
NWDC as agent) shall work with OPNAV, ONR, USJFCOM and other 
services/agencies responsible for these processes to create a 
fully integrated approach to Naval transformation in support of 
joint/combined warfighting capabilities development. 

12. Roles and Responsibilities. 

a. Commander, U.S. Fleet Forces Command. 


(1) Is the Lead Agent for Sea Trial and the supported 
Commander for all Sea Strike, Sea Basing, Sea Shield, and 
FORCEnet CD&E under Sea Trial. 

(2) Is responsible for the composition of Sea Trial CD&E 
Plan (supported by NWDC), review, promulgation and periodic 
modification following STESG deliberations. The CD&E Plan is 
the principle definitive tool for establishing Sea Trial goals 
and will be the basis which overall Sea Trial measures progess. 

(3) Will chair the STESG and act as the approval 
authority for STESG recommendations . 

(4) Will (with NWDC as agent) assign the MUA team and 
associated timeline. 

(5) Will endorse Sea Trial results and direct 
implementation via OPNAV NCDP and PPBE processes. 

(6) FFC N82 is the FFC staff project office for Sea 
Trial, FFC N80 is the overall coordinating authority for FCTs, 
with FFC N82 providing assistance for CD&E efforts. 

b. Commander, U.S. Pacific Fl eet/Commander, U.S. Naval 
Forces Europe. 


(1) Provide forces when available to support the Sea 
Trial Execution Plan. 
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(2) Integrate experiments into fleet exercises to the 
xnaximuiti extent possible while minimizing the impact on fleet 
readiness and training. 

(3) Recommend pertinent/relevant emergent technologies 
and concepts as Sea Trial events. Facilitate efforts to fast 
track select technologies/concepts based on Sea Trial 
assessments. 

(4) Identify to the Operational Agents focus areas for 
prioritizing CD&E, 

(5) Provide representation to the STESG. 
c. Navy Warfare Development Command (NWDC)■ 

(1) Is the supporting commander as the Sea Trial Project 
Coordinator across all Navy/Naval sea trial activities. NWDC is 
the supported commander for development of the Sea Trial CD&E 
Plan and maintenance of the STIMS website. The CD&E Plan and/or 
the Execution Plan may be modified/updated following each STESG 
meeting. CFFC may direct an urgent change at any time. 

(2) will link proposed Sea Trial events to warfighting 
capabilities required to support Sea Power 21. 

(3) will, through representation on all FCTs, coordinate 
with PEOs, SYSCOMs, WCOEs, T&E Centers, labs and ONR to develop 
an integrated and synchronized CD&E; Plan that focuses on Fleet 
priorities and Mission Capability gaps. The OA approved execution 
plans will be synchronized by NWDC and included in the Sea Trial 
Execution Plan. NWDC, reporting to the STESG, will coordinate 
these efforts across all OAs, minimizing (with a goal of 
eliminating) redundancies in experimentation planning, and 
providing an independent assessment of the combined Sea Trial 
process in addressing MCP gaps, ancl other naval and joint 
warfighting capability development requirements. 

(4) Identify for CFFC, via the STESG, areas that are not 
being addressed by any proposed Sea Trial events following 
assessment of the overall Sea Trial, effort. 

(5) Focus the CD&E Plan in accordance with CFFC, 
Operational Agent, opnAV N6/N7, and CG MCCDC priorities while 
consistently working to meet joint requirements. Ensure the 
plan remains agile and aggressively takes advantage of rapidly 
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changing testing requirements and/or opportunities to rapidly 
advance warfighting capability development and deployment. 

(6) As part of the Sea Trial CD&H Plan development, 
ensure the rapid integration of lessons learned, fleet feedback, 
and previous experimentation analysis and assessment. 

(7) Coordinate and liaison with CG MCCDC to integrate 
(where appropriate) USMC CD&E into Sea Trial. 

(8) Supported by the Operational Agents, FCTs, and 
WCOEs, plan and coordinate execution, analysis and assessment of 
Fleet Battle Experiments (FBEs), selected limited objective 
experiments and other experiments as required. 

(9) Promulgate and maintain an experimentation planning 
guidebook to provide execution guidance for field experiments. 

(10) Synchronize experimentation activities and efforts 
to co-evolve the technology, doctrine/TTP and organizational 
changes required to deliver a fielded capability. Recommend to 
CFFC, OA(s) and all other appropriate commands opportunities to 
modify or cancel planned Sea Trial events to eliminate 
duplication of effort, hold down cost, and/or free assets to 
focus on higher priority Sea Trial requirements. 

(11) Leverage other service, joint, and combined 
experimental activities to the maximum extent possible and 
coordinate the fleet's participation in these other experimental 
activities. 

(12) Coordinate all Sea Trial Wargaraes/spiral plans with 
NWC and modify them as required to support Sea Trial CD&E Plan 
priorities. 

(13) Integrate analysis and assessment results into 
DOTMLPF change recommendations for CFFC. 

(14) As operational concepts are validated, promulgate 
them as doctrine/TTP. 

(15) Integrate all tac d&e activities applicable to the 
Sea Trial CD&E Plan. 

(16) Develop and host the interactive STIMS database and 
provide assistance to activities entering data into STIMS as 
necessary. Report to CFFC and all OA(s) on the status of STIMS 
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usage, to ensure all Sea Trial events are fully documented in 
the database. 

(17) Coordinate with ONR, MCCDC/VCNR and SYSCOMs to 
ensure R&D activities support near, mid and long term Sea Trial 
priorities. Obtain visibility over ONR Future Naval 
Capabilities (FNC) and Naval Systems Innovations (NSI) to assist 
in determining which elements are mature enough for inclusion in 
Sea Trial events. 

(18) Apprise CFFC, supported by the Operational Agents 
and all other appropriate commands, of guiding requirements and 
modifications/limitations to approved STESG plans as available 
funding and/or cost estimates to support events evolves. 

(19) Maintain a repository of Sea Trial results. 

d. Commander, Second Fleet (COMSECONDFLT) Commander, Third 
Fleet (COMTHIRDFLT) and Commander, Naval Network Warfare Command 
(COMNAVHETWARCOM) . 

(1) Serve as the supported Commander and Operational 
Agent for Sea Trial activities associated with the pillars under 
their cognizance and as a force provider for Sea Trial 
activities. 

(2) Coordinate fleet activities in their respective AORs 
to support the Sea Trial Execution Plan. 

(3) Coordinate with Marine forces and other numbered 
fleet commanders as appropriate. 

(4) Develop Fleet priorities for CD&E and nominate high 
impact capabilities, activities and technologies to be 
incorporated into CD&E plans. 

(5) Review and approve recommended execution plans 
developed by FCTs in support of assigned pillars. Forward and 
brief issues to the STESG for resolution as required. All 
designated FCTs will fully support OAs in these efforts. 

(6) Support joint and combined CD&E for all pillar 
related activities, as directed, including allied and coalition 
requirements. 
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(7) Provide Military Utility Assessments of Sea Trial 
experimentation to CFFC for approval and forwarding as 
appropriate. 

(8) COMTHIRDFLT coordinate Sea Based Battle Lab (SBBL) 
activities as required to support the Sea Trial CD&E Plan. 

e. Naval war College (NWC)■ 


(1) Coordinate with NWDC to fully integrate and 
synchronize all Sea Trial Wargames/spiral events with the Sea 
Trial CD&E Plan. 

(2) Coordinate, with NWDC, a long-term wargaming plan 
for integration into the Sea Trial CD&E Plan. 

(3) Coordinate with CG MCCDC for integration of USMC 
CD&E into Sea Trial (DIRLAUTH with Expeditionary Force 
Development Center (EFDC) for USMC concept development and 
JCD&E. DIRLAUTH with Marine Corps Warfighting Lab (MCWL) for 
experimentation). 

(4) Coordinate with other services and JFCOM to ensure 
Sea Trial initiatives are reflected in appropriate ongoing joint 
experimentation, and that other service initiatives are 
reflected in applicable naval wargaming. 


f. Chief of Naval Research (CNR) . 


(1) Serve as the supporting commander for Sea Trial S&T 
activities. 

(2) In cooperation with NWDC and MCCDC, nominate FNC and 
NSI activities that are sufficiently mature for inclusion in Sea 
Trial. 

(3) Provide representation to the STESG. 

(4) Provide representation to the FCTs as assigned. 


g. OPNAV H6/N7. 

(1) Serve as supported Warfare Sponsor for Sea Shield, 
Sea Strike, Sea Basing, and FORCEnet in the development of 
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Mission Capability Packages (MCP) /Naval Capabilities Plans and 
follow-on development of a Sponsor Program Proposal (SPP). 

(2) As co-chair of the Technology Oversight Group (TOG), 
and working with ONR, N6/N7 will ensure FNC projects support the 
Sea Trial CD&E Plan objectives where applicable. 

(3) Serve as supporting Resource Sponsor for Sea Shield, 
Sea Basing, Sea Strike and FORCEnet. 

(4) Assign Platform Sponsors as contributing Sea Trial 
resource sources for addressing capability gaps identified by 
both the NCDP and as determined by Operational Agents {Working 
together with OPNAV Warfare Sponsors in a collaborative role). 

(5) Incorporate Sea Trial results as endorsed by CFFC 
into the NCDP or forward to the applicable OPNAV process agent 
for action (e.g. N1 (Manpower and Personnel), N4 (Readiness, 
Material, Facilities), NOOT (Training) ) . Those items that are 
doctrine-related will be passed by CFFC directly to NWDC. 

(6) Provide representation to the STESG. 

(7) Support the FCTs through the consulting membership 
of the various MCP leads to provide a conduit of information 
from the MCPs and NCPs. 

h. Warfare Centers of Excellence (WCOE). 


(1) Integrate tactical development and evaluation into 
the Sea Trial CD&E Plan to include TTP development for 
experiments, TACMEMO development for use of operational 
prototype systems, and schedule events through NWDC in 
conjunction with complementary Sea "rrial field experiments. 

(2) Support CD&E through representation on FCTs assigned 
to support core competency areas. 

i. Systems Commands. 


(1) Ensure appropriate R&D, experimental and spiral 
development activities are included in the Sea Trial CD&E Plan 
and resource participation in Sea Trial activities as required. 

(2) Support Sea Trial activity in coordination with 

NWDC . 
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(3) Provide representation to FCTs as assigned. 

(4) Provide input to CFFC, in conjunction with OPNAV, on 
the capability of the industrial base to evaluate the concepts 
under development as part of the Sea Trial efforts with 
particular focus in meeting MCP identified gaps. 

(5) Provide representation to the STESG. 

j. CG MCGDC (DC, CD)■ 

(1) Serve as a supporting commander as the USMC Sea 
Trial Coordinator and the supported commander for USMC/Navy Sea 
Trial CD&E activities. 

(2) Coordinate USMC CD&E activities, including 
participation in FORCEnet, as required to support the Sea Trial 
CD&E Plan. 

(3) Coordinate USMC Training & Education Command 
(TECOM), EFDC and MCWL activities to include wargeiming as 
required to support Sea Trial CD&E Plan. 

(4) Coordinate USMC S&T efforts and Marine Corps Systems 
Command (MCSC), HQMC and Marine Corps Advocate, and Joint CD&E 
activities in support of Sea Trial CD&E Plan. 

(5) Coordinate assignment of USMC Operational Forces, as 
fleet counterparts, to participate as Sea Trial Operational 
Agents, and assign USMC representatives to the FCTs. 

(6> Provide representation to the STESG. 

(7) Provide input to STESG recommendations. 

k. TYCOMS Provide representation to FCTs as assigned. 

l. Naval Postgraduate School. 


(1) Support Sea Trial through conceptual development 
and specific research in the areas of Sea Shield, Sea Strike, 
Sea Basing and FORCEnet in accordance with CFFC/NPS Memorandum 
of Agreement. 

(2) In conjunction with NWDC, support analysis of Sea 
Trial Event products. Additionally, develop and support long 
term Red Team Cells committed to representing potential major 
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conflict, terrorism and other threats. Red Cells will be tasked 
to support war games and other experiments in order to raise the 
level of Red Team play, and to represent the potential 
capabilities joint forces may have to overcome in the future. 


m. COMOPTEVFOR. is a supporting commander for planning, 
evaluation and operational impact assessment of Sea Trial 


initiatives. 


Distribution: 

SNDL Parts 1 and 2 
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START 




RESTART 
(Update SUMS) 


CD&E Initiatives 


START/ENTER Process via 
STIMS database. Events 
submitted by FCTs, OPNAV. 
ONR, Industry, SYSCOMS, 
other agencies. 

NWDC distribute to ali 
stakeholders. 


Rejected 


OA inform sponsor/ 
CFFC of reason for 
rejection. 


OA VALIDATION V 

• Technically 

fe asi b I e/re levant? 

• Relevant to the 
CD&E Plan? 

• Meet Fleet priority? 

• Meet Mission 
Capability Gap? 

• Appropri ate venue 
identified? 

• Etc. y 


• OA conducts validation 
assisted by NWDC and 
FCT. 


NWDC work to resolve 
or brief to STESG as 
candidate w/o resources. 


Resources 

Available? 


Sponsor develops 
Experiment Plan and Data 
Collection & Analysis Plan 
NWDC aligns events 
across all pillars and 
presents consolidated 
Execution Plan to STESG 
CFFC designates Military 
Utility Assessment (MUA) 
Team and timeline 

o Proposed by OA 


Plan, develop, and coordinate Sea Trial Event 


Conduct Event 


Military Utility 
Assessment 


OA publish Quick look 
Event sponsor complete 
analysis 

MU A team completes 
MUA and submits to 0.\ 
for approval. 
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Significant Enhanced Warfighting Capability 
Impact. 


Enhanced Warfighting Capability Impact. 


CFFC approvc/disapprove and deliver results 
to N70 for entry into Navy Staff 


N80/N82 (NWDC 
assist) 

NWDC track & 
update data base 
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